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ABSTBSCT - ] " ' % 

" The Realization that the facilities available for the 
retrieval of scientific and technical information were being 
underutilized resulted in a project undertaken* by the Erexel 
University School of i Library -and Information Science to develop a 
computer intermediary capable of Assisting u«gfs of scienti f ic . and 
technical information in^the performance of online bibliographic 
searcK^s. Individualized Insttuoticn ior Data Access (IIDA) , the 
computet syste.m developed s dur ing the project, operates' in an 
instructional, and an assistance mode, and can be used tc train users 
to perform online searches or' to aid ifsers in search performance,! The 
system employs a set of diagnostics which ale&t users tc the 
strategic and^syntactic errors they make during the search process. 
^.When IIDA wa"^ operational , a number of evaluations of the system were 
made, qsing .first undergraduate compfr|e£ 4 science majors "and 4 . 

-.experienced online searchers, then library science faculty and 

idents and undergraduate engineering 'students, and finally 
/the Exxon Besearch facility iimplorham Park, New Jersey. 
The evaluation studies, i^ndica/ted that indi vidual& trho < had never done 
online searching were able, using IIDA, to do searches which produced 
satisfactory results. Detailed accounts of the design of the system, ^ 
system operations, and the methodology of the evaluation studies are 
given, and the p£jp ject ^biblip graphy is included. (JL) 
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1. SUMMARY 

Beg inning- in 1976 a project was undertaken by the Drexel University 
School of Library avf Information Science and the Franklin Research "Center 
with the objective of providing a computer- intermediary to aid scientific and . 
technical users of online bibliographic information retrieval systems in the 
performance of data base searches. The project was undertaken o^t of 
recognition that the facilities available for searching and disseminating 
scientific and technical information were beingctmderutilized and that helping 
the people whp would normally be expected to benefit from this information 
would encourage them to become, and to remain, users of these facilities. - 

Technically, the problem of? developing a computer intermediary was 
ofie of nibnjLtoring the conversation between a user and the- information 
retrieval sjrstem, or search service. The complexity of a bibliographic search 
is exacerbated by the searchers uncertainty, at the start, as to what \ 
information exists on desired topic, if any, , Further, the definition of 
"desired topic 11 is likely to change during the search as a Result of probing 
early in the search. Feedback, positive and negative, will affect the user's 
future decisions. Given the degree^of uncertai^r in a bibliographic search, 
a computer program whose function' $U is to assist & user in performing a 
search, and whose inf ormat-ioir is restHc^ted to what it can gain by monitoring 
or by initiating *its own questions to the data base, can not "know" how to jjp 
the search^ It can, at best*, tell when the human user makes a ^stake and it 
can offer general advice on how to proceed. 

4 ■ ■ 1 ' 

Working under these conditions/ we developed a system called 
Individualized Instruction for Data Access (IIDA). IIDA is able both to teach 
a' novice searcher how to search and to assist hinrirf doing so. It is \ 
indiyiduaiized in two HSises: users may take instruction at c a time of their * 
own choosing and their own pace; and, when used as an assistant, IIDA's 
messages to the' user are a direct function of what, t'he user has done in the 
search. Emphasis has been on assistance, always with the assumption that we 
have a relatively inexperienced searcher working in a prdblem-solving 
situatAon'i- . 
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IIDA, as a computer software system, resides in a computer separate 
from that of the search sex^ice. While the original design called for this to 
be a dedicated mini-computer, our experimental implementation was done on a 
large time shared computer, for economic, reasons. Users would send their 
search commands to this computer which wQuld relay them to the search service, 
and then receive and relay the search service's reply. A complete history of 
the search- is then maintained. 

Before beginning a search, an IIDA user has the option of taking some 
computer assisted instruction on how to search. Administration of instruction 
is done by IIDA in its instructional mode . When the u$er is ready to begin an^ 
actual search, he deals with IID^in assistance mode , ^t is in this mode that 
IIDA serves as monitor, relay's messages betweeTl user and search service and 
performs a continuing analysis of the search in progress. .J^IDA makes its 
presence known to the user only upon his request or upon its discovery of a 
problem that implies help may be needed. The heart of the search analysis is 
performed by a,s^of diagndstic procedures, which are concerned with 
everything from fatal syntactic errors, (i.e*. , entry of invalid commands) to 
procedural usages thaft cannot be decisively classified as either correct or 
incorrect, but may be indicative of a problem to come. 
. j 1 Jt 

i 

/Sinc^e the intended user of IIDA is the working scientist or engineer, 
we felt it necessary to evaluate the system's performance in teaching an4 
assisting such persons, and tp do so on a scale. large enough to be 
convincing. Several prelimi«fary\tests were conducted with various student and* 
faculty groups at Drd^l University,, but the main evaluation studies made^ise 




at Drdtel 

ts and «v 



of 100 scientists and mgineers. of the Exxon Research and Engineering Company. 



In brief, the evaluation studies tested the/perf ormance pf IIDA in an 
•industrial setting quite like that envisioned as the apprdprikte* IIDA 
environment. The^. results show that: IIDA trained and assisted users produced 
search results that were satisfactory to them. We can draw the conclusion 
from our, test's that the concept "works" in that it enables thfe novice (to 
searching) end user of scientific 'afid technical information to perform his own 
searches at an acceptable level of competence. 



In this Report, we survey the design of IIDA system (sections 2 
*3 / "5 

through 5), describe how* it operates from the user's perspective (section 6), 

relate* it to other research programs (section 7), describe the evaluation 

program (sections § and 9), project how IIDA an<| systems like it will. evolve 

aniHhat impact they will have in the future (section 10}, acknowledge the 4 • 

h§4rp of many people (section 11), and Tist a bibliography of project-related 

publication (section 12). 
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2. PURPOSE OF I IDA 



^IDA was developed for two purposes. Its original primary purpose 
was to- increase the use of existing, scientific and. technical information * 1 
resources by those who currently do not use them, or un^er-utilize them. The 
second purpose was to investigate the feasibility and _ef feet ivenes's of a* 
particular approach to that objective, which involved research into the nature 
of the activity of interactive bibliographic searching, and development of 
measures of use? performance. 

, v * 

At the inception of the prpject, we undertook to develop an 
operational system to teach and assist end users to make use of bibliographic 
information retrieval systems. This system was- to be operated las a service to 
all interested users. At the conclusion of the^roject 1 s first year, a change 
in priorities at the National Science Foundation resulted in a reorientation 
of the project more toward research into how sfSrchmg is performed, and how 
well a teach ing-a&s is t^nt system could be made to perform, but with no intent 
to* provide more than an experimental version of the system. 

To be more explicit, the IIDA system, in concept, has as it£ 
objective to enable end users of bibliographic information to 'perform their 

Dwi* searches under certain, circumstances, described in Section 4* This was to 

I . . . 

be done by both teaching them how to search and assisting them while 

searching. Inherent in this objective is a limitation on the nature of the 

search objective, generally to something like a few good references, rather 

than an exhaustive bibliography or the single "best" citation on a topic. 

This goal was justified on the grounds that there were a large number 
of scientific and technical persons who were not using or who were under-using 
the resources made available to them (1), often tft high'cost^ and by^the 
assumption that users could do many of their own searches. The result was 
expected, to be more 8 searching overall, and a tendency to apply the time of 
professional searchers to the more difficult cases (2). ' While it is 
impossible to document formally, we note that we felt an attitude of almost 
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hostility on the part of many professional searchers oar intermediaries when we 
first publicly announced our goals for IIDA in 1976, but that recently, say . 
from 1979 on, this idea has seemed. to us^to have? caught on and become accepted 
by the information science community. do not attribute the change in 
attitude solely to our owfr project. More likely, we feel, it is an idea whose 
time has come. 
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\ 3. GENERAL DESCRIPTION OF IIDA 



IJDA is a computer system that serves as an intermediary, between the 
user of an interactive information retrieval system and the system itself. It 
operates in two main modes , instructional and .'assistance. t - 

In the instructional mode , IIDA teaches a user how to perform a 
search, using rather^jprimitive computer assisted instruction. Although there 
are practice searches interspersed with the instructional material, the two 
are distinct — all instruction is pre-use instruction in the sense that 
material is taught in one segment pf the instruction and practiced in the next 
segment. The practice searches, however, are real, 'not "canned," 1 

In the assistance mode , IIDA helps the user in the performance of a 
real search. It is this mode which is the major innovation and the principal 
focus of the research. When operating in this mode, IIDA is an. example of a 
new class of computer assistants or coaches, as defined by Goldstein (3). 

Mechanically, IIDA consists of a set of computer programs resident in 
a computer with which the user .is in communication (see Figure !)• The IIDA 
computer, in turn, communicates with the information retrieval or search 
service, computer (4). Users deal only with the .DIALOG language; IIDA does 
not 'supply its own language or extra search commands. Our original assumption 
was that if the va v lue of an assistant coulji be proven with one representative 
language, it would be reasonable to assume it is valuable for other languages 
of a* similar natury. Near the end of the project, some effort was devoted 
developing some new commands aimed at helping the user find some search terms 
(5)« Several commands were implemented on an experimental basis, and given a 
preliminary test of performance with inconclusive results. i 

/ While conducting a search, the user enters Commands through IIDA, but 

^using the frlALOG language, and normally receives the same responses he would 
if dealing directly with DIALOG. IIDA monitors the conversation, examining 
both commands and responses, looking for errors or indications of problems or * 
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Figure 1. IIDA. GENERAL CONFIGURATION 



potential problems. When any of these is detected, IIDA offers abundant, , 
voluntary help facilities, tailored to the specific problem at hand,, and 
might, for example, offer to refresh a searchers memory on £he*us'e of a 
command in which a syntax error was just made* * 

IIDA is required, then, to monitor and appraise the .performance bf a 
search, and to intervene when a probiem is detected.- Intervention is in. the 
form of: (1) informing the user of the nature of the problem, (2) offering 
advice on what to do next; and, (3) offering a reminder of what reference* 
informatibn is available concerning this specific problem. For example, if 
the error is a syntactic ,one, its nature -will be explained, and the * ^ 

appropriate page number of reference material given to the user. If the error 
were a procedural one, rather than syntactic — for example, creation of too 

many mill sets, advice on proceeding- may 1 be offered as well* 

» *~ ■ 

• «■ ^ 

Because the searches are real, IIDA cannot knotf in advance what the 

$ * *** 

user's search objeQtive is, or the bestt way to" go .abput the search* -Hence, it 
is limited' to observation of what the user does., and what r^&ults he * ' • % 
achieves — it has ' n'o.mor-e informatiqn available, to it than he does, himself* ft 

Obviously, then,' there are few c.Lear "right" or "wrong" procedures, except for ,. 

♦ * s 
the minimal requirement of command syntax* The challenge of designing an. \^ 

ass-ist&nt f ozr^ribl^ographoc^search ih^-1 ies ►therein* 



4. ASSUMPTIONS AND CONSTRAINTS IN IIDA DEVELOPMENT 



IIDA was designed .with a specific user peculation in mind. Although 
it has * some .utility for other groups * the initial design was predicated on the 
*f allowing. user characteristics: 

1. Users have some technical or disciplinary training or education, 
t b in the sense that they are aware of a specialized vocabulary 
within their^wn profession. ^ 

t 

2., Although not trailed* in documentation, they are aWfcre of the 
existence of 4 professional literature, and of the concept of 

citations and abstracts. 

* * ■ — ,._ ^~ 

3. Although not necessarily trained in computer use, they are at 
Jeast not computer-shy, i.e., they have probably used ^computers 
or computer output, or participated in planning computer 
projects, and ^are not made anxious by th^ prospect of hands-on 
use. l ' 

% - 

4* Their objective in using the system is serious, as contrasted 

•with someone who might use a computer system merely to learn to * 
discover its faults* or to "bug" it. IIDA users are presumed to 
be solving a problem of their own when using the system. 

5» Use will be Occasional, perhaps orte to ten times per, year. The 
implication of this assumption is that infrequent users of 
artificial fanguages tend % to forget details, and need frequent 
refresher instruction. * . ^ 

6* Search objectives' will be limited. IIDA users are unlikely to be 
performing patent* searches or exhaustive searches bf any kind. 
They are more likely ^looking for a few ,, £ood M citations, or a 
good general bibliography. 
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5. OVERVIEW OF SYSTEM DESIGN . ^ " 

As noted earlier, IIDA is resident in a, different computer than is 
^ the information retrieval system. JUsers dial IIDA, via Telenet, and log into 
4 its computer, which is the Myltics. time sharing computer at Massachusetts 

Institute of Technology, In the experimental implementation of IIDA, the 
Multics log-in procedure was slightly more complex than that of DIALOG. 
Originally, IIDA was intended to^teside in a dedicated minicomputer, which 
would have made log-in considerably simpler. 

, The instructional programs are resident with IIDA. When the user is 
in instructional mode, he is not in telephone connection with DIALOG, which, 
of course, means that instruction costs less per connect hour than searching. 

' The data bases used with DIALOG in experimental work werfe ERIC, ERIC 

ONTAP, NTIS, and COMPENDEX. IIDA is both data base and search service 

* 

dependent. It would typically require about ^ one man-dsfy of effort to*add a 
^^aew ddta/vbase to IIDA f s DIALOG repertoire, and perhaps three man-months to 
link IIDA to a different search service. The latter requires a complete new 
set of command and response parsing programs, and a different communications 
interface. Each search service might, for example, have a different Way of 
indicating the end. of a message. On the other hand, adding a new DIALOG data 
base merely requires a table indicating the location within a citation record 
of each field. # • ' 

; J .- • ;,. 

, When conceived* of as resident 'in a dedicated minicomputer, ( IIDA was 
planned 'to be able to handle 12 to IS simultaneous users, the exact number 
depending on the* speed and^memory capacity of the computer selected. In the 
experimental implementation, hardware limitations permitted at most 2 users at 
a time, pfteft only", . In a multi-user version, users would have a line 
between IIDA and the search service, IIDA acting then as a concentrator or 
multiplexor. A planned CQmmercial 0 version o£ IIDA, to be produced by the 
I » Franklin Institute Research Laboratories, Inc. will incorporate v .£tyis original 
design feature. 
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5.1 Instruction Mode 

*IIDA f s two operating modes are distinct — a user is clearly in one ot 

the other* The instructional mode consists of two teaching programs, each 

* # r 

followed by a practice search • v. 

The first exercise (6) is a computer assisted instruction program* 
It's purpose is to) introduce the student. user to the minimal set of DIALOG 
commands necessary to perform a meaningful search* The commands introduced 
are: BEGIN, EXPAND, PAGE (needed with EXPAND), SELECT (for single terms - 
only), COMBINE, TYPE, and LOGOFF. All IIDA experiments involved use of print) 
'terminals only, hence the DISPLAY command was not needed. A simplified fontv 
of computer assisted instruction is used to administer this exercise. As each 
command is introduced, the student is asked to type in the command, following 
exactly an example given to him. If it is typed correctly, he sees a replica 
of the DIALOG response to the command. He is expected tp discover for himself 
the gist of what the command does. The full explanation of the command 
follows, rather than precedes, this example, to maximize self-learning. 
Bofedom of the user^was expected to be a possible problem, «and early pilot 
tests/ bore out that it was. —Witty do not, in general, want wordy explanations 
J ( of^something that is self-evident. 

The user is taken, in this fashion, through a complete sample 
sedrch., At its end, if is suggested that he try a search of ■ his own. This is 
exercise 2. Now, IIDA links itself to DIALOGS suggests a search topic to the 
student, and then puts itself into assistance mode. The search topic used in 
experiments was one tried out by the research staff beforehand, and designed 
to be highly likely to result in a success. 

Exercise 3 reverts back, to the presentation of instructional 
material* It covers a variety of advanced topics, including more commands and 
search strategy* Because it would take about one hour to type out all this 
instruction, when a student enters exercise 3, he is given only Che 
introduction to each of the following topics, but he must go through tftis 
miriima^mat^rial : ' . " ' * 
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1. Review of basic commands 

2. Advanced commands 
N*. 3. ^Text searching 

4. Search strategy . 

5» Data base descriptions * 

6. Beginning and ending the search / 

7. IIDA facilities 

The idea behind this approach is to tell the user that information in* these 
topics exists, and -to invite him to explore further at his leisure. A After 
cdfapleting this brief ."reference walk," he is y free to go on to the next 
exercise or to go back to any topic of his choice and explore it in as much 
detail as he wishes. The references walk takes about eight minutes. ,*Phe 
browsing time in the complete set of exercise 3 material is a function of the 
user. 

• * 

w Exercise 4 consists simply of the student performing another search, 

this time of his own choosing. 

Assistance Mode * v 

In assistance mode, th^ user performs^ DIALOG search, and it is 
IIDA's objective to provide assistance to him in doing so. Assistance is , \ 
based on the following principles: 

V 

1. It is,- of course, not possible tor IIDA to "know" how the search 
should be conducted, since each user is the sole judge of the * 
applicability of results to his own objective, and because the 
contents of .data hJses change with time. Thus, even if a 



question has been searched before, *a different searcher uses a 
different data base, and has different perceptions of success. 



2. As a resuit. IIDA is a reactive system. It cannot tell the user 
how to perform the seatch, but it can detect various errors and 
poor usages, and can offer general* advice on how to proceed, in 
the context of the error discovered. i\ 
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In o,rder to detect er.rors and poor usage, IIDA employs' a set of 

diagnostic procedures, which are" grouped into three classes: 

<■» • ^^^ ^ 

a* Syntactic * IIDA will detect any error in DIALOG syntax* 

IIDA provides the user as much information as possible about 
the nature, of the. error, e.g., not simply the notation 
"syntax error, 11 but a clear statement of what part of the 
input' command is acceptable and what part not, including what 
IIDA jfcssumed to have been the command* For example, if a 
user entered the command TYPE 3/6/2, DIALOG would assume. this 
to be a TYPE command, from the initial letter* It would then 
expect the next character to be a set number and, njft fipding 
one, would declare the command invalid* IIDA would st^te the. 
assumed commjajid-f^and state that from "YPE 11 on, th^ argument 
of the command is invalid, and that a set number was expected 
after "T." Neither IIDA nor DIALOG tries to correct the - 
error* 

b* Procedural Error — Single Command * This group of diagnostics 
is concerned with commands which pass the syntactic analysis, 
but, when examined in the context of the history of the 
search to date, are. found to be faulty to ^some extent* The 
faults can range from fatal, as in the case of an invalid set 




Abe* (eig., COMBINE 3 and 4, but set 4 has not been 
^r-d^ftaed) , ' to minor (e*g., repetition of a command)* 



. Prodedural Error — Strings * These diagnostics are concerned 
with strings of commands* * A strifcg is defined as an unbroken 
series of commands of the same type* 'An example of such a 
diagnostic is a test for string length* If a user enters . 
more than five consecutive COMBINE commands , a diagnostic is 
"triggered," which'need dofno mor£ than to suggest that he 
consider doing something different* 
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4. Each tim^e a diagnostic detects a fault: 

a. A statement is sent to the user'describing the faulty This 
message is terse and emotionally neutral. 

' h. A record of the occurrence of the fault is made search 
history ,£ile. This file is then used for interpreting later 
commands. It will, for example, contain a list of numbers of 
the sets created, and hence provides the basis for deciding 
whether a Bet number in a* command is valid. 

c. In many cases, the user is offered "quick advice," i.e., a 
suggested action based on the type of error made, not the 
specifics of theVommand. An example is that after an$ 
excessive dumber ot COMBINE commands has triggered a string 
lengtlr diagnostic, the advice will be to try printing some of 
the records to get an idea of their contents, rather than 
defining so many new sets. f 

" d.' In all cases, the Ujjer is referred to a reference number in 
the exercise 3 library of instructional material, which 
identifies a page or frame of tutorial material relevant to 
the error just made. For example, if a syntactic er^or has 
, beeji detected, the reference will be to a frame describing 

. ' the syntactic rules for this commatld type. 

The complete set of diagnostics-is documented in Quarterly 
Report 4 (-7). A discussion of how well these performed, when 
viewed as a measure of user performance, is found in (8). » 

5.3 . Help Facilities ' 

Like most, on-line systems, IIDA has a/^ELP facility.- The user may 

v ;<- . f 

enter the command /HELP* at any time. He is* ^ then given the option of entering 
the help library (actually the library of frames for exercise 3), or viewing 



* FolLowing a commoti programming pr^tice, we use* tHe character M /" preceding 
a command to distinguish this use of - the word "help 11 from any other use. 



any of several summaries of his, search so far (commands used, sets created, 
records viewed, errors made, or terms used), or of returning to the search or 

to on4^ of the exercises. Quick advice, described above, is also a help option 

_ ., — , % 

* 

A shorthand notation, /HN, where n is a page or frame number in the 
reference library, enables the user to go directly from a search to* that 
frame, without first calling for /HELP and seeing the hel*p menu of services 
offered. ' , v ^ > 

In addition to the option of returning to the search, a user may exit 
from HELP at any time by entering a valid DIALOG command. , 

5.4 ( Post-9Earch Summary 

Going back to the original design concept for IIDA, it had been 
planned that j IIDA would operate out of a dedicated minicomputer, witfi up to 
12-15 simultaneous users. To support this operation, there was to be a 
proctor who would be located physically near the computer, and accessible to 

4 

users either by message through the user's terminal or by telephone. The 
proctor would be charged with providing whatever help a user needed, ranging 
from coping with the mechanics of the terminal to advice on search .strategy. 
To assist the proctor, he would be supported by a high speed terminal, and 
would have access to the history files of any user. Thus, if] called by a 
user, or if called by IIDA because of an excessive number of diagnostics, the 
proctor could quickly review the history and status o£ tne search <23). 

1 

When we implemented IIDA in its experimental form, because of , 
hardwire limitatidns, it was never possible tor more than 2 people to use it 
at one time, and our experimental procedures were such tljat these would always 
be working through our own termij^ls under controlled conditions. Hence, we 
did not need the proctor. However, the history tables developed ^.around the 
idea of a proctor were still thought to be useful, and proved to be > so. 



They could be used as reference aids by searchers, through the help 
tacility. This option was seldom, if ever, invoked iti our tests. They proved 
invaluable to the IIDA research* staff iA preparing data for statistical 
analysis. Examples of the tables- are shown in (9). 

13 ' . 



From a computer software point of view, the main components of IIDA 
are:* a communications interface, (2) a parser, (3) the diagnostic 
procedures, (4) the Warning control program, (5) the instructional programs,- 
(6) the help programs, and (7) the post-search summary. These are summarized 
below, and 'are explained in greater detail in Quarterly Report 6 (10), The 
approximate relationship among the programs is illustrated in Figure 2. 



5. 5*1 The 1 Communications Inter f ace . • The purpose of^tl\is program is 

» 

to handle the passing of messages in either direction between the IIDA 

computer and the user's terminal, or the search service computer. *>The Multics 

computer has an automatic dial-out device, called an autocall, which initiates 

* — ' 

the outgoing t^eleph6ne* call from the computer to the TELENET of TYMNET 

networks, in order to reach DIALOG. Incoming messages frt>m DIALOG pose a 

particular problem because they arrive in fragments, and do not contain an 4 

explicit end-of-message indicator. Hence, the program is sometimes not 

certain when a message, rather than merely a packet, has ended. For example, 

a Jong abstract may consist of several packets, and the IIDA communications 

interface gets no explicit signal from DIALOG That the abstract has^ended. . A 

timing £evice can be used, but sometraes network delays. made it 'appear that <* 

the end of message has been reache{$when it has not, and the result is 

cutting^ the user off in mid-message. Certain messages have known endings 

(e^g. , the response to an EXPAND ends in "-M0R&- 11 ), ^rft A the end of a response 

to a TYPE command cannot always be detected. Thus, the success of a system * 

such as IIDA depends upon its ability to keep the flow of messages moving 

smoothly, and to inform the user when ohere has been an interruption or delay 

beyond the capability of IIDA to control. Jr~^ 

5.5.2 The Parser . The parser must«parse, or separate into syntactic 
components, th£ user's input, the search service's responses, and the 
network's occasional messages that may interrupt the flow of user-search 
service traffic. Dn messages from the search service or the nfetwork, the 
parser recognizes the component parts. On messages from the user, the parser 
also checks for syntactic validity. Although it is logically desirable for 
parsing and validity checking to be separate functions, it is not always ^ 
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possible, because recognition of succeeding syntactic elements depends on 
. p-receding elements* Thus validity checking must go on as the parsing goes on, 

Hr. _ 

'.Parsing is highly dependent on the .search language and on the 
composition of messages received from the. search service. Changes in language 
or in search service responses, such as are Qccasioned by introducing a new 
search service, make for major changes to IIDA, while changing the system to' 
accommodate another data base offered by the search service for which 
programmed i« ^ relatively minor problem, because only the data record 
structure changes, not the commaind language or repertoire of response messages* 

Both the parrser and the communications interface, as actually 
•implemented, were heavily dependent upon software developed by Marcus and 
Reintjes for the CONIT project (13) . v 



5. 5. 3 The Diagnostio Procedures . Onpe the parser has broken a 
command and its response into'their elementary syntactic ^components flnd added 
these to appropriate tables describing "the history % of the search, eact\, 
diagnostic rule, a^ software, is little more than a complex PL/I IF 
statement. The design principle here was to use the diagnostic rule (IF 

,s tat ement— RULE has about the same meaning it ha$ in the decision table 
context) to determine what -rules were violated, and then, in a separate 
action,' decide' what to do about ifr,^pr them. ^Oth the parser and the \ 
diagnostics use and maintain the search history files.. ' j * 

5.5.4 The Earning Control Program . A diagnostic rule signals 
warning control that a rule has been violated. Warning control decides what 
messages 'to send to the user as' a result. In its initial Resign, warning % 
control 'performed two pri4e functions. First, if more than one rule was 
violated by a single command or its response K warning control decided to which 
rule violation to respond. The general principle was to respond to 4 the ifrost 1 
specific. For example, if a User violated both a dwelling (too many similar^ 
COhfBINE commands, with sinjil^r results) and a' string length' rul^ with the same 
command, warning control would respond to the dwelling rule violation, as the 
more spec if iq, hence probably the more valuable for the user to know about. 
'The second functio^of waging control was to be obn<;ernea with the frequency 
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>^>f diagnostic messages to theWiser (or .warning messages, hence the name). In 
some cases, a rule was not violated until some error had occurred more than n 
tynes, for' example the generation of a null set. Generating a null set does 
not, in itself, trigger a diagnostic reply frdm IID/f, but generating several 
of them* does. On the other hand, once a warning or diagnostic message is 
given}\nd the error rfecurs immediately thereafter, the designers felt there' 
would to no point in immediately repeating the di^iostic message. For 
example^ if a ^tring length error occurs, a diagnosfTc^ message is sent, but if 
the user adds another command to ^the same string,*there is no point reminding 
him again of the string lenfeth problem. Instead, IIDA would suppress messages $ 
responding to this particular error until^ the condition recurred m timtfs. The 
variables n and m were given arbitrary' values', with the intent that 
experimental experience _would lead to better settings. If, after a given rule 
had- been suppressed once, it tKen was ^permitted to be used, i.e., th$ 
recurrence frequency m^iad been exceeded, the message would be enhanced witb 
some reminder that the user had seen this message before. The objective v/as 
to erihance without nagging., 

A latfer version^ of warning control made the resetting of thresholds a 
variable, t and made available a command for the user to reduce the frequency of 
occurrence of diagnostic messages. If the user gave the command/SLACK after 
any diagnostic message, then the threshold for the* rule that produced that 
message would be changed and made less stringent. This would enable a user 
, who was yiolat\ng IIDA rules but felt he knew what he was doing to proceed 
witbout interruption. In our experiments, this facility was never used, and 
we now deem it to have b§£n elaborate than the situation calls for. ** 

5.5.5 The Instructional Programs . The instructional programs of 
exercise I and 3 are a simple form of computer assisted instruction. In 
exfercise 1, the.re is no .variation of the sequence of instructional frames. 
All students are taken through the same- material in the same order. Users are 
asked to enter commands ,\ but they are to enter a command exactly as provided 
by the instructional tame preceding. In the event of any error, the user is 
askefd to re-enter the>command. This approach was taken in tftm interest of 
quick implementation, since the^CAI was not the focus of our experimental 
work, but ^fet'was deemed important to, have it in order to test the entire 
system concept. 
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In exercise 3, the user has a choice of material (except the first 
m *^ime through, when he must view only the major topic description^) . However, 
no questions^bout substance are asked. The facility is primarily used, after 
the quick "reference^ walk/, M as a voluntary reference tool. 

5.5 . 6 The Help Program * Help is almost a control program. Upon 
invoking HELP, the user is gi\>en_an optio^h of changing IIDA modes, i.e., 
entering into ajiy exercise, stacking or restarting a search, or of directly 
going to a message deemed by the diagnostic programs to be relevant to the 
most recent command and its result* In the latter cases, ma^. diagnostic 
messages offer a reference number in *he exercise 3 library of instructional 
frames that is specific to the error just detected. Also, in many cases, 
quick advice is offered, again in terms^ of a pre-stored message deemed by jghe 
diagnostics to be relevant to the current .situation. Thus, HELP, as a 
program, is really one which switches tne user to another program of choice. 

Again, in the, interest of b^oth development and program' running time, 
rto attempt was made *to provide for IIDA to directly* answer user initiated 
questions. Rather, the user is told where information may be found, an<J he is 
H then expected to find it. 

5.5.7 The Post-Search Summary . As noted earlier, the post-search- 

. summary is merejS^ a printing out of, the seardh history tables used during the 
conduct of the search to assist the diagnostic process, and were originally 
^ planned for use by a 'proctor. • * 
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6. HOW I IDA OPERATES 

« 

s. 

The first-time user of IIDA generally begins with exercjse mode in 
* order to learn the basics' of on-line searching. After learning, the basic 
commands and becoming acciistomed to the system, the user selects assistance 
mode and performs his DIALOG search in the ordinary way,' except that* IIDA 
informs him of errors and offers an extensive 4 HELP program which is available 
dpring the search. The user is irfStructed on how, to use the LIDA facilities 
(the HELP program arid 'diagnostics) during the tf^ercise or trainii>fe phase'of 
his ItDA expedience. Each time the aser logs qn^Jie* is offered a review of ' 
the ^commands and procedures unique to IIDA., 

•/ • • * ■ ■ 

6,1 Introduction to IIDA * 

The urfer'ft first session begins. with afik introduction to* the terminal 



and login procedures. During dew^jg£ment and 'ttffrt'^pg^ assifl€^€? were on hand 
to get use|$ started. '^*The aasis^nt d£|^sd the local TELENET o^ TYMNET 
number, initiated IIDA's identification procedures, >and completed* necessary 
protocol to connect the user to the Multics' ^y.stem. During this activity, the 
assistant instructed the- user in how to send a message "to the -computer" if 
-the user had had no experience with terminals. The user was then ^given a 
personal identification number, and was instructed to "enter your number whan 
IIDA asks for it." The user was then on his owfiXfcith IIDA., 



x 



The instructional exercises of IIDA are structured iq individual 

*■ . ' * % <» 

frames of information of varying lengthy After eacty frame, the user is asked 

to enter a command, answer a question, 4>r otherwise sfet}d a message t6 IIDA. 

The useir,' after completing login* procedures, is offered introductory 
frames on the IIDA system. Pour frames,/ which explain the purpose of IIDA, * 
some basic information on on-line searching, the components of the programs, 
how to correct ah in£ut erron, and the use "of the HELP program, are available 
to* the user eac^time he logs on. He may see the entire introduction i>y 
answering "YES" to the qUestion, "Wobld. you like more information about % IIDA?" 
which follows 'the first three frames. Upon completion of the introductory 
frames, or upon answering "HO" to any of IIDA's offers df introductory 
material, the user is showtv a me^ii of ojiC3»ns and is a$ked' to select, one. * 



To begin one of the following enter its number: 

. Exercise 1; basic DIALOG commands ' 

Exercijse 2: practice DIALOG search 

gxejr^ise 3: advanced commands and search strategy 

Exercise 4: search assistance only 

Enter '/done' if you are finished with IIDA. 

I?* 1 * • 

The user enters the^ number of the exercise or mode he wishes to 
begin. He .may exit the program altogether* by entering the response/done. 

r 

6.2 Exercise Mode ^ 

'Exercise Mode consists of three parts: exercise 1 introduces the 
user to seven basic DIALOG commands and the IIDA HELP command; exercise 2 
provides an opportunity for the user bo conduct a single search using the 
commands learned in exercise 1; and exercise 3 presents additional detail on 
the use of the basic commands, an introduction to advanced commands and 
practices used by experienced searchers, and brief descriptions of the data 
bases available through IIDA. An in-depth presentation of the use of the IIDA 
facilities and of search strategies is also included in exercise 3. 
-\ 

6.2.1 Exercise 1 .. Exercise 1 uses a simulation of a search in 
COMPENDEX (file 8) to teach the basic DIALOG commands. The . "canned" search is 
on the use of solar energy tt> heat residential buildings. The- system 
introduces the user to basic concepts of on-line searching, presents the 
search topic, and then tel>ls the user what he is to do during the exercise*. 

Jfe l re now going tp take >you through a simple search. The 
subject of the search is the use of solar collectors or 
y v heat pumps in residential btfildings. You'll be told what 
to type. Type in exactly what you're told and then you'll 
see what OlALOQ does with that command. L&ter on, you can 
' do t a search o£ m your own. 

* - * 

Enter BEGIN 8 

D? begin 8 4 
V** ^ 21nov79 11:40:51 User 9002 
$0*18 0.012~Hrs Filel* * 
Filed: COMPENDEX 70-79/AUG x 
' (copr* Engineering Index Inc.) $ 

Set Items Description (+ a 0R; *"AND;-»N0T) 

• % # « * * 
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From the initial BEGIN command to the end of the search, IIDA instructs the 
user to enter commands and to view the system's response. Following each" 
response, IIDA describes briefly what each command meant and. what the 
responses mean in terms* of the search as a whole. 

♦ 

By the end of exercise 1, which takes about 20 minutes to complete, 
the user is familiar with the DIALOG commands BEGIN, EXPAND, SELECT, COMBINE, 
TYPE, PAGE, and LOGOFF, and with IIDA's HELP command. 

Upon completion of exercise 1, the user is offered the menu of 
exercises and modes, and is free to choose an option or leave the system 
entirely. 

6.2.2 Exercise 2 . Exercise 2 is a practice search which enables the 
user to use the commands and techniques he learned in exercise 1. Two search 
topics are introduced. These are offered specifically as both subjects are 
extensively covered in the data base; IIDA's intent is to assure the novice 
user considerable success in his first search 



Exercise 2 allows you to practice searching on DIALOG. 
It is possible to perform a successful search on a rath'er 
specific topic using only the six basic commands- learned in 
Exercise 1. Practi££vour search skills by finding - ** 
citations on one or thk following topics using COMPENDEX 
(file 8)?. ' 

1. Asphalt paving for road and bridge surfaces 

2. Passive solar energy used to' heat residential 
buildings 

4 

In, this practice exercise you should try various approaches 
to searching while directing 'your efforts to the goa^ls 
specified above. In the future, determine before beginning 
to search whether you are looking for a few relevant items 
on the subject or whether you, are after an exhaustive > 
biblibgraphy. 

Take a moment to think of some terms which are -likely 
to appear in citations on the topic you have chosen. Note 
terms which you will use in developing your search. For 
example, you might plan to EXPAND the term 'solar 1 to see 
terms useful in a search on 'solar heating 1 • Also consider 
EXPANDing the term ' asphalt 1 for terms which may be useful 
in a search on •asphalt paving. 1 / 
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The user is asked to specify the numtbr of itema he wants to retrieve; this 
information is used' later by diagnostics concerned with search results* 

The user is told that "IIDA will place^ a telephone call to the DIALOG 
system" and that procedure will take a minute or two* . . 

IIDA handles the entire logon procedure without, input from the user* 
owever, 'because the procedure does take a few nvinutes to complete, the user 
is informed of the initialization and status of telecommunication processes, 
the fact that the DIALOG login procedure has been completed or aborted, and 
whether or not another attempt to connect to DIALOG had b£en initiated* After 
an unsuccessful attempt to login to DIALOG, the user may leave- the system 
entirely or try again immediately* If his attempt is ^successful, flDA tells 
him to start his search by using the BEGIN command with the number of the 
appropriate file* 




\ 

Logging onto DIALOG _ . 

News from DIALOG: . 
wwwwwwww LOGON 'Pilel Tue^0nov79 16:25:32 Port04E ) 
FILES 3 and 16 ARE OFFLINE ... 

DIALOG AVAIU REGULAR HOURS DURING U.S. ' ( , 

THANKSGIVING HOLIDAY BERIOD. , 
REGISTER OflLINE 'FOR BEST 'OF UPDATE IN \ 
% LONDON ON DECEMBER J • SEE ?NEWS. % 

NEW DISCLOSURE (FILE 100) NOW AVAILABLE \ 

WITH HALF HOUR FREE CQNfiECT-TIME IN NOV " , 

WHITE HOUSE CONF. HOTLINE^(FILE 202). \ x 
Yoii are now connected to Lockheed DIALOG retrieval systetia— 
Choose data basis 1,, 6, 8, or/ 201 with the BEGIN command, 

D? beginl s 
20nov79 16:25:59 tJser 7303 
$0.14 0.009 Hrs Filel* v . 

Filel*;EMC 66-79/OCT # ' * % 

« Set Items Description (^OR^AND^^NOT) . 

The user carries out his search in the usual manner, and IIDA is 
available as an assistant through 'the HELP program, and as a monitor of .the ^ * < 
messages the user sends. If a syntactic error is detected, IIDA tells the * 
user that an errotf was made, and gives the frame number, for the 'HELP program. -i "* 
The user is not forced to use fiELP', as t many errors are apparent upon receipt 
\ x of the' error message. IIDA also asks the user, to rate the relevance of 

.citations typed out. this rating <xs used for several purposes: fxrst, IIDA 
converses with the user on his progress in terms of relevance. Secondly, IIDA , 
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can inform the ijper that his search may be nearly complete if, at some point, 
he has typed a fair number of highly relevant citations. Finally, the rating 
of the relevance of individual .citations forces the user to think about what 
citations he has retrieved, and to look for concepts that he has not included 
in his search* In .essence, the user is dperafc^fig as if he were in assistance 
mode in exercise 2; the instructional features of the exercise are that the 
user is prompted to think abou.t his topic and appropriate search terms before 
accessing DIALOG, he is given a choice of vetfy specific topic's to searcli, he 

is forced to consider the citations retrieved in terms of relevance, and he is 

c 

given another chance to learn by doing. 

6.2.3 Exercise 3 . Exercise 3 serves two purposes: first, it is an 
instructional program, and secondly, its contents comprise the HELP library 

which may be accessed when the user is searching in assistance mode. 

» * » » 

When the user selects exercise 3 from the menu of exercises or modes, 
he first sees eight frames of information. The first frame introduces the 
user to the exercise, and the remaining se$en serve as a table of ^contents for 
the remaining 88 frames of information^ The seven categories of instruction 
are listed in the illustration below; 



Contents of Instructional Units 
a (1) Review of Basic Commands 

- begin, expand, select, page, combine, type, 
logoff 

*b (2) Advanced Commands . 

- file selection, limiting, type and print, 

* abbreviation, time-saying techniques, search 

sav^ - 
c (3) Test Searching 

- searching for \^>rds or phrases embedded in 
larger sentences or paragraphs 

d (4) Search Strategy,, 

- planning the search in advance 
e (5) Database Descriptions, 

- contents* and use of data bases 
f (6) Beginning and Ending the Search 

- how to initiate^your conversation with DIALOG 
'and * how to sign off when you 1 re finished 

g (7) IIDA Facilities 1 

\- * how to use IIDA.during your search 
* " • ' 

Strike the carriage -retuxiuwhen you are ready to see the next frame* 
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The numbered frames (1) through (7) are mandatory for the user in exercise 
3 — he must see them all before g^Jlg to further detail on any of the 
individual topics outlined* Following the eighth frame JfraiL (7) IIDA 
Facilities], the user is given the option of continuing with exercise 3 or of 
beginning a search in assistance mode. 

4 

You have seen the introductory frames of Exercise 3. 
You now have two options: 

(1) <to see more detail on any of the topics listed^above, 
enter the two-digit frame number preceding the topic 
of interest. 

(2) To begin your search £n Assistance Mode, enter 
'/done 1 . Enter a letter, any frame number, or /done. 

If the user elects to continue with the exercise, he enters the 
number of any frame introduced by frames (1) through (7). The frames may be ^ 
viewed in any order, and the user may follow a topic in as little or as much 
detail as he feels he requires. Each major topic is divided into subtopics 
which provide increasingly detailed instructions on the use of commands and 
techniques. When the user has exhausted one line of instruction tow£he most 
detailed level, he is instructed to enter a frame number to begin a new topic 
oP instruction. During tfte exercise, the user is instructed to enter various 
commands, just as he'was in exercise 1, and to view the system's response.. 
When all frames have been seen, o?r when the user has seen all the information 
he thinks^ he needs, he exits exercise 3 by entering /DONE. 

If the user chooses to begin searching in. assistance mode after 
seeing the eight introductory frames, he may access the instructional material 
by using the /HELP command, by selecting HELP option #1, during his on-line 
search. The content of the frames is the same as what Che user in exercise 3 
sees; the only difference is that the user does not, in assistance mode, have 
the opportunity to practice the commands and techniques in simulated steps. 
Since the usee in assistance mode is in the midst of a search when he accesses 
the HELP library, he> is usually ready to return to the search as quickly as 
poffsible; use of the table of contents of exerdise *3 and of the error messages 
encountered during the search, which indicate which help frame to access, 
allows the user to have near direct access to the necessary flELP frames. The 
user\n assistance mode returns to the search* from the HELP library by 
entering a DIALOG command. ' ^ 
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Complete transcripts of exercised 1, 2, and 3 are included in C6)* 

* * " « 

6*3 Assistance* Mode 

Assistance mod$ is selected by^ttie^fiser when he is ready to^p.erform 
an actual search. After logging in, the user 'selects assistance mode from the 
menu of exercises and modes* The automatic connection? and login to DIALOG 
described in 6*2, is carried out, and the user is asked to enter the number of 
items he wants to retrieve* Unlike exercise 2, however, there is no 
introduction to assistance mode^from the user's view he is "on his own*" 

** - 

When. logged in to DIALOG, the user performs hxs search m the usual 

dinner, beginning with the BEGIN command* Diagnostic messages may be sent if 
errors are detected* These messages frequently contain reference 'to the HELP 
fr^me where correct usage is explained* The user may or may not choose to 
access the suggested information, but message syntax must be legal to be sent* 

As long jas the user is getting. productive results and asks for no 
help, I IDA* remains in the background until the user performs a TYPE command* 
Then, as in exercise 2, IIDA^asks the user to # rate relevance of citations 
printed (TYPED) out^ A scale of 1 (hot relevant) to 5 (highly relevant) is 
used* IIDA averages^the relevance rating of sets of items viewed, and, over a 
sequence of TYPE commands, informs the user whether his assessment of the 
items retrieved is improving or declining* The user can modify his strategy 
•to retrieve more relevant material^* If the relevance comment average 
vdeclines over the typing of ^ev^eral sets, IIDA mAy offer assistance in 
correcting the problem, .either at the user's initiation or^through a strategic 
ertbjc Message* 

6*3*1 User-^nitjated HELP * 'There are three types of user-initiated 
HELP J reviews of the search from various perspectives, quick advice, and the 
help library. Either may be called* for by t^e user any time he neecls 
assistance* „ ~ 
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options; 



By entering the command /HELP, the user, receives a menu of HELP 

(1) the hIlP library 

(2) command history # 

/v * N 

(3) set history 

(4) history -of records ^iewed during the search 

(5) history of errors- made during the search 



(6) history of descriptors, 
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From this menu, he may select any* option he feels will assist in his 
search. The histories (options 2-6) are designed to help the user keep track 
of the search in progress, and to allow him to see the search in perspective. 
For example, the history of descriptors used may prompt the* user to bring a 
forgotten term into the search. 

Once the user has seen the /HELP menu, he may use help directly by 
merely entering /H and the number of the HELP option desired. For example, 
paving seen the menu once, a user might enter /H3 to review the sets created 
in his search. 

A* second HELP option is available through the use of .anpther IIDA 
command, /QA. /QA stands for ,f qui.ck advice," and provicfes the user with a 
brief recommendation from IIDA as to an approach to take. This advice is 
based on the histories maintained through the IIDA program, and lets the user 
know what approach anight be appropriate following the most recent command 
entered. The QA option can also provide reassurance in that one of its 
messages tells theuser tKat no advice is needed at the time. * 

The thixjd user initiated HELP option is the library. It consists of 
the 80 instructional frames that, comprise exercise d. HELP is invoked 'by the 
user by entering the command /HI, or by selection of option 1 from the /HELP 
menu* When the user/enters /Hi, he receives the menu of . seVen broad topics of 
exercise 3, described in 6.2.^ At the end of the menu, the user is asked to * 
select the topic of his choice by entering the number or letter of* the topic 
selected. Since the frames of instruction are structured in a * "* 

general-to-specific tree arrangement., * the user may pursue the topic selected 
"to whatever degree otf detail necessary. 
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To exit 'the HELP library, the user simply enters a DIALOG command to 
return to the search. The user may pursue as many frames a$ needed while in 
HELP. The only dif ference^Wween using HELP library while doing a search and 
using the frames as an instructional exercise is that when in a DIALOG search 
the user does not have the opportunity to practice using new Commands and 
techniques — they are printed out as examples within the .Instructional frames. 

6.3.2 IIDA-Initiated Assistance . Because IIDA maintains extensive 
histories on all aspects of the search in progress, the system is able to 
provide advice on strategic and syntactic errors. 

One program describes syntactic error conditions and certain "hybrid 11 
conditions which are neither syntactic' nor strategic (for example, including a 
null set in a COMBINE command). Approximately 100 error messages are- 
reconstructed by the program, which tell the user whatikind of error he has 
made, what IIDA has "assumed, 11 and finally 'the reason the command is illegal^ 
or invalid. The messages also inform the user of what to do to get further 
help. For example, a message might read; % 

* « - . 

D? type F8/5/1-4 ' ■ 

* 80*** SYNTAX ERROR: Command assumed: 'type 1 Argument" 
assumed F8^5/l-4. The argument is not acceptable because 
it 'does. not begin with a»set number. For help, enter 
/HI. 13^ ,m m 

The user may proceed with the search .by correcting the error, or, he may take 
the /HELP -option offered and review the material in the library. 

. •> 

"v. - / 

Strategic tgerr or analysis is less generalized. The content and *> f 
context of 1 each user message is analyzed and compared with previous search 
history, ''if strategic errors, or unproductive conditions, based on IIDA's 
rules,' are detected, the program determines if a message is to be sent to the 
user. Evep strategic error does not invoke a message, since some .flexibility 
helps the user to experiment with different approaches, and too many "helpful"- 
messages could irritate and confuse; the novice searcher. 
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IIDA offers assistance on the following types of strategic errors: 



(1) creation of null sets . * » » 

(2) low relevance ratings or decreasing ratings ^ \ 
.(3) use 'of uninfoTmative TYPE formats 

(4) creation but nonuse of sets ^ ' * 

(5) convergence and dwelling conditions " e 

(6) repetition of commands 

(7) x pn>blems with string length: excessive use of a sirigle command* 

consecutively 



The user receives a message as to what occurred, why, and what to do to 
correct the condition or to, get help. For example: 



30£Set examination: the following sets were given a relevance 
rating ^0^*4 or more 6ut were not typed with a full format when 
examined • 

2 • — " . 

6 ■ . 

8 (.. ^ 
You may i!rifilj^to resume this 'search to examine these sets in a full 

' ' / format. * 

The user is free to/follow -HDA's advice or to ignore it. IIDA f s 
role is to inform e the *user ojf the strategic & condition, offer help, and to 
allow the user the freedom tto use a^strabfegy comfortable for him as a 
searcher. IIDA's help is available as the user needs it, but recognition is 
given to the fact that w variations in searching logic and style exist, even 



among novices andjBcasual users 



s 



\ 4 
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7. RELATIONSHIP OF. I IDA TO OTHER SYSTEMS 



While IIDA is essentially a new system, many of its features are 
related to philosophies, approaches, and techniques developed elsewhere. The 
purpose of this section is to trace the influences of concurrent and previous * 
work on IIDA's development. k ' • 

• ) ; • ' 

t / NLS Scholar. (11) 'is arr artificiaj intelligence based system -designed 
to teach non-programmers to use the NLS editor, part of a modular system use^ . 
as an aid in writing, editing, indexing, and publishing. J,ike IlbA, NLS 
Scholar combines instruction with actual use 2>f a. system which performs 
certain defined tasks. NLS' Scholar' s emphasis i& on the natural language 
interface, and instructions^ practice, and online; assistance components are 
provided.. The major difference between IIDA and NLS Scholar lies in the 
difference between online bibliographic searching f which is iftore structured 
t handed itingV^and the loose structure <y| the editing or writing .tasks of NLS 
Scholar. v -IIDA can identify and assist in handling user errors rather 
\ specifically without • the user needing to- inquire of the syst6m as to what Co 
ao next or how to correct a problem. / • **'««. ^ 

"SOPHIE, b Sophisticated Instructional Environment (12), is a computer 
assisted instruction system .which utilized artificial intelligence in the 
teaching of electronic circuit analysis and troubleshooting* Like" NLS 
Scholar, SOPHIE provides instruction which is tailored, to the individual; not 
only can the user progress at hia own rate, bat the systems provide advi,ce to 
the user which is based on what he is doing, in the context of a real ^ 
problem/ Becalee the systems recognize that there are a variety of acceptable 
approaches to each task undertaken by the user\.the user'is encouraged to find 
the most suitable approach within the ^road scoRe of "correct " % 

troubleshooting," "effective editing," or Vpro.ductive online bibliographic ' * 
searching." C 



SOPHIE differs from, IIDA in several aspects. Most importantly, in 
• SOPHIE th^re is always a "correct solution to the users 1 problems. When the 

IIDA user performs his search in DIALOG, there is no singLe measui^ of. + 

performance since the search i# structured and the relevance of the results 

• » . * 
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evaluated by the user himself. IIDA has, in a sense, extended the range of 
possib/e approaches available to the user because the DIALOG search is real in 
every sense, not a simulated situation. Because of the nature of the task 
involved, SOPHIE offers simulated circuit analysis and troubleshooting. 

• . . . / 

CONIT (Connector for Networked Information Transfer) is a product of 

the M.I.T. Electronic Systems Laboratory (13), and like j^gCDA was designed to 

V. . . . . . • . 

make it possible for an end user of bibliographic information to perform 

, \. * 

online 'searchfi^ without the help of a trained intermediary. ^CONIT provides 
its own command language which the end user utilizes to 'search several data 
bases supplied by more than one commercial vendor. IIDA, on the other hand, 
teaches DIALOG and offers the user the limited choice of four fij.es — ERIC* 
NTIS\ COMPENDEX, and ONTAP ERIC. CONIT translates its, contmand language into 
that x>f the selected data base vendor's language, enabling the use? to learn 
only one language for use with multiple services. On the other hand, IIDA 
users 'learn DIALOG'^ commands. The commands^- ±£_vaJJLd, are sent by IIDA s 
directly to DIALOG. Therefore t^^£^i s n0 translation from IIDA 's .language to 
DIALOG'S — the^ are the same with the exception of's few commands which invoke 
the IIDA help program and which allow the user to change f rom .oYie IIDA mode oc* 
exetcise to another, or to exit the IIDA system altogether. * 

'A second major difference between thersystems is wi^h regard to, their 
assistance programs. CONIT and IIDA both offer extensive online assistance to 
the user with a searcfi in progress. CONIT's Assistance is more general in - 
nature than IIDA's. .The IIDA help programs evaluate the user's errors in 
syntax or strategy in the context of the specific search in progress. IIDA 
maintains anxtTxtensive history of each search which is analyzed when- an error 
is made^or when the user_ asks for advicte. The help is rendered on the basis 
'of tKe user's errors as well as strategies and techniques assumed to lead to 
productive searching b^ ^asual or novice^u£ers. 

CONIT assists the user in selecting the appropriate data base fot his 
search.,. IIDA, on the other hand, gives a raord limited choice of data bases 
and only explfains them to the user in exercise 3. 
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Another system which predates IIDA is the University of^ Pittsburgh 1 s 
CAI TRAINER (14). This system has some similarities to IIDA's instructional 
mode. TRAINER, like IIDA, teaches online searching and has a "practice" mode, 
but the important difference is in the form of practice. In IIDA*, the user' is 
connected to the .DIALOG information retrieval system in the second exercise 
and performs an actual DIALOG search. TRAINER offeVs simulated searching from 
three files. 

* * 

0 

While the CAI contents bear some similarities, these are as they - * 
pertain to actual searching* IIDA teaches the user, several commands [HELP, 
/QA (Quick Advice), etc.] peculiar to IIDA, while TRAINER instructs in 
terminal u#e, telecommunications protocol^ identification, access procedures, 
and even offers practice in losing in. Since IIDA handles tile login 
procedures for DIALOG, there is no reason to instruct the user in that process 
in an extensive way.^ 

TRAINER, viike IIDA, is designed for the casual users; however, it . 
provides no interface with the actual search systems and was designed 
specifically to perform the training function. 

■ ■ ^ • r 

The work of both Penniman 015) and Standera (16) influenced the 
apprpacb which IIDA t^kes in teaching the search process. Penniman ^ 
Categorized search' commands into four groups : indebc search, logic 
formulation, document display, and other administrative commands. Standera's 
observations led to the identification of a slightly different classification 
of commands: start^stop; execufTLvd (logic and display commands); supportive 
(index searoh^ set history); informative (procedural qr data t&se 
information); and auxiliary (offline print, copy ordering, etcO. 

IIDA, for instructional purposes, has used Penniman^s categorization, 
slightly modified by Standera 1 s: 

. (1) Index search (EXPAND; SELECT w/single term) " 

(2) Logic formation (SELECT w/full text capability, COMBINE, etc.) 

(3) Record display (TYPE) * • J ^ 
-n (4) Procedural (BEGIN, LOGOFF, FILfe), 

(5) .Diagnostic (HELP, EXPLAIN, etc.) v 

4 



y The first three of these categories also define the order of * 
proceeding with a search tKat. is taught by IIDA tutorial exercises. Once 
dding their own search (e.g., exercise. 2), the users may modify the order tx> 
suit personal style. One concept that is introduced in IIDA is that the users 
are explicitly introduced to the notion of recycling through the index search 
and logic formation commands should the record display* reveal that the set 
created on the first cycle needs to be refined or re-defined. ' 

IIDA's aproach in teaching only the seven basic command (BEGIN, 
SELECT, EXPAND, PAGE, COMBINE, TYPE, LpGOFF) and then encouraging users to 
practice "real" searching, is in keeping with Standera's view t;hat most, 
searchers confine themselves to very few commands in the performance of an 
online bibliographic search. Exercise' 1 of IIDA was designed to introduce 
online searching to the user and to get 4 him starred. Exercise 3, largely 
cipt'xonal as an instructional program, provides advanced commands and 
techniques. 



-35- : . : 

• •» ' 8. EVALUATION »; ' 

• • . 

In the evaluation of IIDA,. the concern has been'both fcith formative 

and summative evaluation* In formative evaluation, the focus has been upon 
i < • 

monitoring, syst^jn' development and s with providing feedback. and information * 

leading to the refinement and further development of the system* For example, 

a number of the diagnostic rules in the assistance mode require specification 

of a threshold value* Initially these values were set by intuition or 

arbitrary choice* However, the use of the system has enabled revision of some 

of Jthe values towards more appropriate levels* 1 

• In the^summative evaluation of IIDA the fconcern has been with 
asseasment of the impact and effectiveness of the IIDA system and with the 
degree to which th$ objectives of the project have been met* In the 
discussion which follows the three major phase*s of, IIDA evaluation are 
summarized* The fir$t- &nd second-stage evaluation are described more fully ; 
in (*17), and a description of the details .of the third-stage appears in (18)*^ 

8*1 * First-Stage Evaluation 

Although labeled first-£tage evaluation, the kind of formative 
evaluations described here were.to some extent conducted continuously 
thtoughout the entire development of. IIDA* ~ ~~ " 

• ^ 

8*1*1 Project Staff Evaluation . v Once exercises 1 and 2 had been* 
made operational, and on a continuing basis thereafter, the various exercises . 
were" subject to review for improvement and modification by all members of the 
project staff* In. addition, all members of the staff spent time interacting 
with IIDA ab users- in order to look for flaws or *yays to improve the opera$ipn 
of the system* . *\ 

i . • ~ 

8*l*2 t Computer Science Majors * Another group which worked with the . 
system during this' early testing were three undergraduate. computer science 
majors who acted as' users 'and who were challenged to find the problems* The 
.emphasis in this testing was upon exercise 2 which not only stands as part of 
IIDA but which also, in a sense, provided a nucleus which was expanded into 
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the* assistance mode. All three of the students were able to search 
successfully using IIDA and ran intp very few problems in doing a search. 
Those programming and technical flaws discovered at this stage were 
subsequently corrected. 

> \ m 9 

» 4 1 s 

8.1.3 "Real"' Searches . V^Lth exercise 2 fully operational, a number 
of searches done by real searchers ^were re-done through ^this exercise in order 
to look at the responses of the system and the diagnostics to "real" 



searches. ««Ihe sevent^^wo Searches involved were taken from a set collected 
by Fenichql (24) in a study on search process assessment. Every member of tKe 
project staff regardless >of their role on the projct duplicated a minimum of 
four searches through exercise 2. Thus, all members of the project staff were 
involved in looking for flaws that needed to be corrected. In addition, all 
of the staff members were able to gain first-hand experience with how IIDA 
would have treated the user who had done the search. 

While no major substantive changes were immediately made in the 
system as a result of this exercise, it was of particular benefit in insuring 
that the entire staff had .a coherent and complete feel for the project as a 
whole rather than for just that sub-component upon which each happened to be 
wording at the time. Purthermdre, it was also of benefit in helping to 
provide a basis for later decisions about .various questions which arose about 

9 e 

such' issues as how and what to', implement in the way of messages from IIDA, 
etc. Finally,^ it provided various staff members with*"aTr4mportant background 
•context against which to weigh various user comments collected at the time of 

use or during post-use interviews and discussions. 

J 

8.1.4 Library and Information Science Faculty and Graduate 
Students . Three graduate students and t;wo faculty members of the Drexel 

University School-of Library and Information Science also went through the 

I 

entire set of IIDA exercises (exercises 1 through 3) before testing*was begun 

i 

t # i # • 

with noyice users. Otjce. again these, tests were useful m that they -uncovered 
programming errors which had not yet been discovered and corrected. • All of 
these users were experienced with online bibliographic searching and generally 
liked the IIDA programs*' However,, they were not completely successful in 



41 



detecting problems with the content of the instructional programs from the' 
novice user's point df view* In later testing, undergraduate engineering 
students with no library training or bibliographic searching experience found 
difficulty both in comprehending some aspects of the material in exercise 1, 
and in absorbing the bulk of material in exercise 3. Apparently it was 
•difficult for the library school test participants, as for th % e staff, given 
ttfeir familiarity with online^ searching, <o appreciate the way a complete 
nQvice would respond to the instructional- material* 

8.2 Second-Stage Evaluation 

2 — : : * K 

In the second-stage of the process of IIDA evaluation, there were two 
studies conducted which provided information on both formative and summative 
evaluation. Each of these two studies was addressed to a different kind of 
qfiestion. Given the structure of the total IIDA system, it is possible to 
look at the effects of IIDA when the system operates only as an assistant. 
When dealing with this question, it is assumed that the user haKprevious,ly 
had a reasonable amount of search training and engages IIDA only through the 
assistance mode. .The second kind of question one can ask has to do with the 
effectiveness of the IIDA exercises in tesching new users to do bibliograhic 

informaticrti retrieval. When dealing with this* issue, it is assumed that the 

* ,< 

user has had no previous direct experience with searching and utilizes the 

«• 

capacities of IIDA as both instructor and % assistant. 

8.2.1 IIDA as .Assistant . * Each year the School of Library and 
Information Science at Drexel University admits a number of new graduate 
students. When these ^students are admitted, they are required to register for 
a course" entitled, "Fundamentals of Library ffnd Information Science." On^of * 
the cpmponents of this course is a block of instruction in computer based 
bibliographic searching.* This block of instruction, roughly totaling thirteen 
hours,, includes both classroom lectures and hands-on laboratory experience in 
searching. TheT Fundamentals students were recommended as a group for study 

only by their accessibility., but also because they knew how to search at a 

. , . . • f* 

novice level, jrere fairly homogeneous in training and experience, 'and could be 

expected to experience some problems with their searches whiclTwbuld trigger 

±hi diagnostics'. - v ' * ^ - 



.^From a summer session class of twenty-six students, there were 
twenty-two volunteers* Of . these twenty-two, Seventeen actually began and 
completed the study. Each student was asked to conduct two' searches in the ^ 
same order. Randomly, half of the users were assigned to conduct the two ' 
searches with IIDA assistance while the other half were assigned to' do the 
searches without assistance. In both cases ,the' searching was done through 
IIDA so that the search records were kept automatically, but with the no-IIDA 
group the interactive messages were suppressed so that the users were nob 
given the' IIDA feedback. Five .sets of search records were lost as the result 
of system difficulties which. did not appear to be statistically related to the 
user's group membership^ This loss left five users in the no-IIDA group and 
seven users in the IIDA group. 

Comparisons o£ the two groups revealed that they did not differ 
statistically from e^ch other in terms of the frequencies with which the 
various categories of diagnostics were invoked during the process of 
searching. The data used were those from the second' search only. The only 
differences between the two groups found in examination of the attitude 
measures on the post-search questionnaires showed up on two questions. While 
the IIDA assisted group showed a higher level" of agreement with the statement 
that the assistance mode of IIDA had helped in searching more effectively, it 
wa^s the no-IIDA group which indicated the highest degree of agreement with the 
statement that the IIDA searching had been enjoyable. 

~~ - The users described in this study were all trained butt inexperienced 
searchers, some of whom did searches with IIDA assistance. In general the 
reactions of those who did their searches with IIDA were positive in that they 
felt the system had provided th^m with valuable or worthwhile assistance. 
While there did not appear to be any effect of IIDA assistance^upon the 
frequency with which the diagnostics werp triggered during the search, the * 
number of searches in each group was relatively small and so there may have 
been effects which ^were not detected. It was reassuring, however, to find 
that the direction of differences wpuld have favored the IlDA group if t\ie 
results had been significant. This would make it reasonable to assume tha£^ 
even if IIDA assistance did hot fieSrs^irably improve* the searcher f s performance, 
it also did not seem to impair it* 



V 

8.2.2 IIDA as Instructor . "Turning to the issue of how well JIDA is 
able to perform the service of enabling novice users to do a successful ' 
search, the concern is basically with a user group which has had no previous 
experience, either direct or instructional, with computerized information 
retrieval. One such available group of users was found among Drexel 
undergraduates enrolled in courses in technical writing. The study originally 
planned for this group was abandoned as the result of* unanticipated hardware 
problems. A Hpwever, a number of> students were provided with IIDA training and 
search assistance. Thus, the study described below was descriptive rather 
than experimental in nature. 

In all, thirty-five undergraduates, mostly Junior and Senior 
engineering majors, were recruited from two summer session sections of 
technical Writing. Student participation in the IIDA training was strongly 
encouraged by the instructors, but participation was «not a class requirement. 
The students were encouraged to use the opportunities for searching to compile 
bibliograhies for their term paper and other writing assignments in the 
course. These users were initially scheduled to do exercises 1 and 2 in one 
session and exercise 3 and a search of their own choice during a second 
session. Various kinds of difficulties resulted in many of the students 
having to return for onp or more additional sessions. * 

Of the original thirty-five students who began, thirteen dropped out 
or did not finish the three exercises and searching forgone reason or 
another. Based upon user comments, instructor feedback, and a post-training 
debriefing conducted with one of the -technical writing classes as a group, the 
high dropout rate was attributable • to two major factors.* The first of these 
was the length and amount of material* covered in exercise 3 as it was 
structured for these users.' The second factor contributing to the high 
dropout rate was a relatively high frequency of system difficulties external 
to the IIDA programs. While occasional hardware problems were encountered 
with the computer that housed IIDA, more often there were problems with DIALOG 
going down. However^ the single moVt^frequent set of problems arose with 
telecommunication network difficulties which resulted in several participants 
being, recj&ired to schedule more than two sessions to complete the cburse of - 
training and searching. Some users simply refused, others just disappeared. 
There were also a few who dropped the writing course and drppped IIDA as well. 

.' ' ~* ■ ' 44 
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^ Although 'this study provided no comparative data on the diagnostics 

for different user groups, it did result in an intensive examination ,of the 
operation of the machine scoring procedures* Because of the considerable 
online interference encountered during much of the study, some "errors" 
reported by' the computer were not caused by the seai*ch£rs* «fhe search* 
transcripts were examined in detail by the IIDA staff and the jnactiine reported 
errors were analyzed to, determine which were in fact a result of a user 
difficulty. Those faults or errors which resulted from a malfunctioning of 
the IIDA programs were brought to the attention of the system programmer so 
that future ^problems could be minimized or avoided* 

v 

One of the questions on the post-training questionnaire asked the 
students to report the percentage of items retrieved in the search that were: 
(a) Very Useful, (b) Useful, and (c) Useless, Since the ultimate criterio^ 
for the value of the information retrieved lies in the judgment of the user, 
this was felt to <J?e a particularly important index tcf the success of the 
system* The first two categories were combined into a single percentage 
useful score, with 'an average of 60% of the references retrieved being judged 
either useful or very useful* Not surprisingly, there was some variability in 
this score, with extremes ranging from 0% fo 100%* If was also the case that 
the users 1 judgments of the percentages of useful items retrieved were 
significantly correlated with some of the other variables investigated* 
Percentage useful was negatively correlated with the number of syntax errors 
made aa<J* was positively correlated with scores on various questions .designed 
to assess the users 1 attitudes towards different aspects of IIDA and 
IIDA-assisted searching* This pattern" is suggestive in that it raises the, - 
possibility that problems with syntax errors inf luenced^fche degree to w{iich 

^ it 

information was successfully retrieved* It further suggests the possibility 

that the users 1 - evaluations, of the search system, when doing their own 

searching, may be influenced by the degree to which useful information is 

1 ~ 
retrieved* * / 

J* 

In terms of their detailed comments about the' system, almost all of 
* * 

the users in this group found positive things to say* They also found 
negative things to say* In addition, these users suggested a number of 
detailed kinds of ideas about- things they did not find in IIDA that they 
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thought should be included, and they were able to provide .detailed feedback 
about a number of instances' in which they felt that IIDA had done ot said the 
wrong thing, F^ver than* one-third of thes£ users felt that they would be able 
to do searching on their own without IIDA assistance, 

« 

* 8,2,3 Discussion , It seems, from the information presented so far, 
that by the^end of the second stage of testing the IIDA system was in good 
enough shape to be .able to provide the type of training and assistance 

^necessary to enable novice users (at least some novice users) to do online 
bibliographic searching*,, Certainly there is no reason to expect that the 
'technical writing students would have been able to^sit down at a terminal with 
no prior instruction and no online assistance and be able to search as well as 
they did. Thus, one of the # major goals of the project had been abcomplished, 
i,e,, the development of such a ifys'tem irkia computer-based interactive * 
framework. It was also clear, however, that a'number of substantive problems 
remained in that wTiile the average percentage of useful items retrieved by the 
technical writing user group was 60%, there were several users who were 
dissatisfied with '^the system or were unable (or, possibly, unwilling*) to 
r&trieve any reference which they felt to be useful. Had the evaluation of 
IIDA ended with the work reported to this point, a reasonable summation of the 

♦ results of the project wpuld have been, "It seems to work, .but major 
improvements remain to be nAde," 

The suimner studies* were , however, pilot testing for subsequent field 

experiments to be reported on below. Several lessons were learned from- the 

* • 

use of IIDA by the various user groups which led to major benefits in the 
conduct of the field studies. One obvious benefit from the pilot testing Jtay * 
simply in the use of the instructional and assistance programs with real users 
searching real problems. This led to the discovery of as yet undiscovered 
, technical and programming problems. In addition, the 'testing, along with the 
network transmission and other technical problems encountered during the 
summer, motivated a> detailed search by -search comparison between the search 
transcrinfcs produce^ 'by users and the machine kept records of search 
activity! The results of this comparison led to improvements in the operation 
of the automatic record keeping functions of the programs, 

* si . * * 

* . 1 i 
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Soke additional' benefits which accrued frdra the summer pilot testing 

> / 

involved the questionnaires and the research assistants. The extensive 

*»»\ 

fee&b^ck from the technical jrciting students also included commentary about 
the questionnaires used in the study. This feedback, and examination of the 
data actually collected on the questionnaires, led to modifications in, and a 
considerable shortening of, the questionnaires. The research assistants 
involved in the project gained invaluable experience from the pilot testing in 
dealing with the problems of working with the system, in working with the 
users being trained, in coping witjv the problems of scheduling users, and in 
conforming to the needs of a research plan. The assistants were also able to 
develop a number of important insights into the operation of the system and 
the problems of the users. x 



\ Also as a result of testing experiences with the various user groups, 
especially with the technical writing ^tudents, modifications of the 
instructional exercises were introduced to either clarify the information or 
to make the exercise more palatable. The - types of changes made to exercise 1 - 
were primarily those of clarification. For example, at first some students 
expected to reprieve actuaj. facts about their topic rather than references. 
Others "did not seem initially to realize that the commands in exercise 1 were 
to be learned for future use. Still others displayed a conceptual 
misunderstanding. At 'the time of the summer testing, exercise 1 employed an 
example siearch on Library Automation, as a format 'for introducing the various 
commands 4 and illustrating their use. At one point in the search, IIDA asked 
the user to enter EXPAND* LIBRARY. A few of the usexfs were not entirely , clear , 
however, about how to us.e EXPAND, as illustrated by the fact that they - 
subsequently thought they should enter EXPAND LIBRARY no matter whatGhe term 
was that was to' be expanded. '"All of. these problems wer£ spotted early in the 
testing and appropriate changes to exercise 1 were introduced. 

* < ~* 
After the completion of the training of the technical writing 
students there was also an extensive revision of the structure of exercise 3. 
In its original format,- th£s exercise consisted of eight frames with extensive 
discussion and detailed elaboration of the material in each, as a form of 
advanced training. Since each user; was required ta go. through all of the 
material, ix^ the exercise, the total time required to complete it averaged 
about forty minutes. The exericse was Changed in that the structure o*f the 
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program was modified to require the user to 1$ exposed to only th^P , 

introductory descriptions of the various advanced training materials. Thus* 

the user is now required to take a refereftce walk through exercise 3 in^rder „ 

to become familiar with^the contents of the advanced training materia^ Then 

the-user has the option either of learning more in exercise 3, on a 

self-selected advanced study ba&is_, or of going on to the assistance mode. 

Once in the assistance modfe the user has the option of returning to the 

contents Qf exercise 3, in the form of the help library, whenever necessary. 
^ ■» 

One important point which was convincingly demonstrated by the 
second-stage evaluation of IIDA is that the kind of pilot" testing described 
here is an indispensable part of the development of any system. as large and 
complex, and as user-oriented as IIDA. Because of their closeness to the 
material, the system designers are, at times, not the best judges of what 
material should be viewed by, explained to, or simply available to the user. 
It is very easy to let the necessary concern with the prcfblems of technical 4 
development 'outweigh the seemingly more mundane problems of user 
compatibility.' There is, however, in a user-oriented system, little to 
commend the most technically brilliant programming accomplishment or design 
feature if it is beyond the capability of the user to take advantag^^it. 



8,3 . Third-Stage Evaluation 

The third-stage of IIDA evaluation following- closely upon the 
completion of the system revisions suggested by the results of the 
second-stage evaluations. The studies described here were conducted with the 
'kinds of users, and in *the environment, for which the system was intended. 
During the fall of 1979 and the winter of 1979-1980, two field experiments 
were* conducted with research engineers and research chemists at two different 
Exxon Rtfeearch and Engineering Company sites in New Jersey. In one of th'ese 
studies~the baseline study — the performance and reactions of IIDA trained and 
assisted searchers were compared^Tfct^ t^ose^of end users who had had^ searches 
done for them by the professional intermediaries at the Information -Center on 
'site. In the second study—the training method study 4 — the performance and 
reactions of IIDA 'trained and assisted searchers were compared with those of 
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searchers who were given more conventional classroom-type^ training and then 
assisted in searching by IID£. (T^hus, in one* case the focus was 'upon the 
adequacy of IIDA-ds a total syst^tn^or enabling the user to retrieve 
information, and in the other the fAcxxs was Upon the IIDA training materials 
as a way of providing online search training* 

8*3.1 The Baseline Study . The baseline study was conducted at the 
* — 
Exxon Research and Engineering Company facility at Flbrham Park, New Jersey, 

in the fall of 1979. The study design called for collecting data on the 

searches performed by twenty-five- searchers who wete trained and assisted by 

IIDA. In addition, twenty-five searches Vere to be done by the information 

retrieval staff on site. 

The design # of the study was a simple treatment and control group 
design with half of the participants being randomly assigned to each group. 
The fifty participants were randomly selected** f rom a list of almost* 150 
volunteers who had been recruited by a mailing which described /the opportunity 
to beJLnvolyed in IIDA bibliographic search training and offered them the 
opportunity to participate. Those participants assigned to the IIDA group 
were trained to search using the IIDA programs and then did two searches on 
subjects of their own choosing. The other participants, in the intermediated 
group, submitted the next search topic arising from their regular work to the 
information center intermediaries ♦ These searches were then conducted by the 
intermediaries through IIDA, but with the IIDA diagnostic messages being 
suppressed. 'All searches were done in Compendex. 

• Si „ « 

As the result of an IIDA program system difficulty -some of the search 
records for the intermediated searches were not preserved and* printed out for 
later use. Of the topics actually submitted, IIDA diagnostic data were 
available on only sixteen of the searches. * CoosequentTy, the data that were 
available ,on all of_th£ intermediated searches, e.g., the order in which the 
searches were clone and the post-search questionnaire questions, were split 
• into tyo group.s. • One group consisted of those participants where the 

diagnostic data had been Lost, the othet the group where the diagnostic data ' 

>- — 

had been preserved. Statistical comparisons using the conservative alpha s 
level of «25 revealed no significant differences between theses/two groups so 
it was assumed that the loss of search transcripts, was effectively random. 
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^ Comparison of > the participants in the IIDA trained and intermediated^ 

groups pn several prersearch measures revealed no significant differences % 
between the groups, indicating that the randomization had*\iad the desired 
effect. -These two groupsralso did not differ from each other on several 
post-search measures of tneir reactions to the searches. For example, in 
terms of judgments of the percentage of useful litems retrieved in the 
searches, the difference was net statistically signif icaat, vlt^ the IIDA 
trained .group indicating an average of 52.5% of the references retrieved were 
useful, while the intermediated search participants" indicated that kn average 
\J>f 49. 3% of the references Retrieved were -useful. Thus, the IIDA trained and r 
assisted users Were able to produce satisfactory searches. It was also found 
that there werertto significant differences between the two sets o'f searches in 

terms of the ^number of IIDA detected errors or faults in the searching 

« * a'' 

behavior of the. professionals and in the searching behavior of thg IIDA 
assisted users. *Althotigh\compfiirabJ:e infprraation Wars not available for both 
* groups o'^uwg, the ^responses rof the IIDA trained and assisted, users to 

open-ended evaft^ioi^upstions seemed *to clearly indicate an overall positive i 
response to the sjfet^V " Thia^iS J$$t tjjf say^ however, that there were still ) 
not things that the tfders' <fid^^ improvement in* the 

system from second- to thirds tage^fevafS^^ri is cleanly Suggested by the fact 
that 818% of the IIDA trained and asdis tea Ofbrtf *wei^ef willing' to recommend the 
online stearch system, Jfccltjding IIDA, ^thjajb: friends* In addition, 88% of, . . 




th&se users 'felt able to search on.^the\^ owqJ T^ls represents a considerable 
change from the response of the technic&L wrJu.j*g, srtudenfs involved in working 
with IIDA in the £econd-s£age evaluati 



In light, of -the' interit'of* this study, 'i.p*., }dT provide a set of 



^Jt ; diagnostic 'benchmark criteria against which to assess th€ performitf^e pf the 
kF^-~^*~ IIDA^trained and j^ssisted searchers, it w$s surprising to. discover a>iack of 
significant 'differences. between the two groups pf searches*^ A significant 
difference on one or more of the diagnostics wpuld ftave pointed toward a p . t 
deficiency in the IIDA training or in the usefulness of the diagnostic % ^ ^ 
' * messages during IIDA assisted searching. T^e fitfding of no significant 

differences also poses * t potential problem in thaiLit is not cleefr whether jf . 

this means that the IIDA diagnostics worked Well^ or that they are not J 
sufficiently sensitive to the type of "errors" detected. * \ / 
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In looking at the lack of differences between the twb _> ^et§lof. 
searches, there are some circumstances which suggest that the professionals 
may have been somewhat handicapped relative to their habitual mode 
^eartfhing. For example, the Exxon professional searchers are used to working 
with 1200 baud terminals. With these terminals, more online printing is v 
acceptable than with the 300 baud terminals i^ed for IIDA searching. Hence, 
t , the professional staff was accustomed to doing a great deal of online 

printing, often exceeding the limit allowed for IIDA users working with the 
• slower terminal. Also, IIDA's design was fixed just; before Lockheed announced 
the new su^er select command (25). Hence, this command was not included in 
the* IIDA training materials and its syntax' was not recognized by IIDA's 
parser, and would trigger a diagnostic error. Consequently, although" it iff* 
not possible to assign numeric values, there is reason to believe that the 
error-fault ratej>f the professional (searches might have been higher than 
normal. 

T 

In addition, the professional intermediaries at Florham Park weWj a> * 

^ — r» 

working under unusual tume constraints due to severe scheduling problems • 
These time constraints made it difficult for professionals to searcV^in thei^ 
"normal" manner. The scheduling of the intermediaries 1 searches on IIDA 
^fecame such a problem that, part way through the experiment, these searches 
were begun on IIDA, bu'fc were then saved, executed, and printed on ExxoaJ^s 1200 
baud terminals. 

Another factor to be considered in Evaluating the results of this 
experiment is the possible difference in the nature^pf the searc.hes conducted 
by IIDA users and the professional intermediaries. The professional searchers 
performed comprehensive searches on a pre-defined 'topic. It was- not required 
that engineers 4jave a pre-defined search topic or that they do a comprehensive 
search. There is considerable doubt that tjie engineers $nd intermediaries did 
the same kind of searches. Nonetheless, it is certainly true that the-HDA 
searchers performed respectably well and were satisfied with their search 
results. 
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One reason for arguing that the. training and assistance routines 
worked as intended lies in the fact that the end user evaluation of the - 
utility of the information reflk^eved in the searches did not differ, 
significantly between the two groups. IIDA users also did not differ 
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significantly from those participants who ha\i had intermediated searches done 
for them in terms either <of their degree of sd^sf^ction with the search 
systtem used or of their willingness to recommend the search system to their 
friends • 



8.3.2 The Training Method Study . The second major test of IIDA took 

/, — ■ : 

pftace at the Exxon Research and Engineering Company facility in .Linden, New 
Jersey, during the winter months of 1979-1980. This study involved a % * M 
comparison of the searches done by IIDA trained and assisted searchers with 
those done by a group of searchers who were also- assisted by IIDA but who had • 
been trained in a two-hour classroom training session. The intent of this, 
study was to test out the instructional procedures and materials involved in 
*IIDA training against a more frequently used and more conventional training 
method. w ^ 

The design of the study was a simple two. treatment experimental 
-design with half of the participants being assigned randomly to each group. 

As in the baseline sXudy, the fifty participants were randomly selected from a 

« _ t. 

list of almost 150 volunteers who had been recruited by' a mailing which 

described the opportunity to be involved in bibliograhic search training and 

«* ^ ' 

offered them the opportunity to participate. Those partici pant s assigned to 

the IIDA trained group were trained to search using the IIDA programs and then 

' i* 

did two searches on -subjects of their own Choosing. The other participants, 
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in the conventionally trained group, jdid two searches of their own choosing, 
with IIDA assistance, after having been trained at one of four training^ ' 
sessions .organized and taught hy one of r the Information Center professionals. 
Although each session wak conducted by a different professional, the .same 
instructional materials and handouts were used in each. The experimental 
design did not allow for online practice time,/ but some participants saw an 
online ^demonstration. All searches were done in Cotopendex. j 7 

Comparisons of the two groups on several pre-search measures revealed 
only one significant difference between the two groups. s This difference, in 
the average number of computer languages known by the users in the £wo groups, 
with the IIDA group having a, higher average, did not seem to^be a very strong 
one, howevep; in that further investigation indicated that the number of users 
who knew no languages at all was not significantly different, between the two 



groups' and that those users who knew one or more computer languages did not 
differ between the two groups in terms of ttte mean number* of languages known. 
Furthermore, the number of languages known turned^out not to be correlated 
with any other variable investigated in the study except the individual's, 
self-rating of- computer skills, and the two groups did npt differ 
significantly on this measure* " ^ 

The two grqups also did not differ from each other on several %t 
post-searcti measure? of their reactions to the* searches and the search- 
•assistance system. For example, in terms of i the judgments o£ the percentage , 
of useful items retrieved in the searches, the conventionally trained users- 
indicated ^tnafe\ 62. 9Z of the references were useful and the' IIDA' trained users 
indicated tha#, on the average, 46. 6% of the references retrieved in their 
searches were useful. This difference was not statistically significant. 
Where differences did exist, they were related to characteristics of the + t 
search assistance system rather than the search results.* Those users who were 
conventionally trained seemed to.be more likely than the IIDA trained 
searcherSx^o agree that the instructions ^in the search system w^re^easy~to f * 
follow- and t|iat the systfem was easier^to yse ,than had been expecte^oT^ In * 
addition, thei conventionally tratned/group expected to encounter more 
difficulty with the system than -wa^r in "fact the case. Interestingly enough,^ 
these difference's reflect favorably on the IIDA a<ssistance programs in that-* * 
experience wirtr'the IIDA ttfaihing exercises is clearly npt necessary for the 
individual to' be able to make use of the assistance mode. 

It was also found that there were no, significant differences between 
the two se.ts of searches in terma^ of the number of IIDA detected errors or 
faults in the searching behavior of tfre, two groups.. Finally, although .> 
statistical analyses were not conducted, the responses of the two groups of 
searchers to open-ended evaluation questions seemed to' clearly indicate two 
things. *' The first is IIDA is not a finished product without fault. The 

questions designed to elicit negative comments from the users wete clearly 
able to do 'so. The second is that overall most of the users seemed to like 
learning to search, either with or without IID^, and. they seemed positive 
about the IIDA assistance they received. It is also interesting to observe 
that the users did not draw a clear distinction between the charactervStis* of 
IIDA and those of DIALOG or the telecommunications. This is not surprising 
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since IIDA was designed to be ad unobtrusive as possible during searching* „ 

However, system designers should be aware of the fact that users generally 

tend to evaluate the system as a whole and that the newly designed component, 

such as IIDA, may get the credit or the blame for characteristics of the pther 
v , « 

/ K 
parts of the system* % * 

The finding of no significant difJ^erentes on the diagnostic measures 

y 

could p^pse .a probleta for interpretation of the results were it. not for the 

. * * «. * * 

fact that both groups were able to* retrieve a significant percentage of useful 

references during their searches*^ Presumably this represents something which 

neither group would have been able to accomplish if simply turned loose with a 

terminal but without having any training or assistance of any kind* . 

Furthermore, the two groups did not di£fer significantly in fheir estimates of, 

the percentage of useful references retrieved* In addition, a tot^l of ^84% of 

the* participants in each of t±e groups were of the opinion that they would be 

able to search on their own, and even more were willing to recommend the 



system to th^tr friends, with 92% of the conventionally trailed g^^p and 88% 
''of the IIDA trained gjroup, checking this" question jLn the affirmative* 

This pattern of results argues strongly for^ the "idea that IIDA 
training, as it is presently structured, represents a viable, alternative to 
the type of conventional training with which it was compared* One reasonable 
guess as, tp the reason fbr the pattern of results reported here is that while 
the hirman^tnjtruc tor may well in fact have been more flexible and responsive 
during ,trainin]g*in 'assisting the. student and in answering questions, the 
design of -IIDA^does, as intended, ' enable the user to discover where and how to 
get the information needed to answeri* the questions for .himself* IIDA provides 
an avenue of access to online searching for individuals who, .could not or would 
got do a conventional training course and who still wanted personal rather 
than intermediated access to a data base* . . 
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8*4 Genera}. Discussion ( * ,* 

A major goal of this project If&s been to\prpvide%a rifetftod for 
allowing direct user access to bibliographic searching* Thus, the attempt has 
been to develop 4 a set of computer software packages which, can provide online 
assistance to occasional users of information retrieval systems. This 
collection of programs is also intended to be abl€ to provide instruction, if 
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needed, in the ^commands used in* searching 'and in search strategy. Whejf 
originally conceived, the expected utility of IIiJA lay in the area or what 
might be .called "problem solving searches •". Theste are searches where th$ end 
usfer of the information does not know exactly what the characteristics are of * 
the desired set of references until^they haye* actually been found. 
Consequently, it is- very difficult for the .end user to describe the problem to 
an intermediary. There is no reason, however, why the IIDA user could not and 
should not make use of the system for all kinds of searches, if desired. 

The primary intended IIDA user has been the working scientist or 
engineer who may need access to the data base only a few times a year, and 
consequently is not interested in training oriented toward those who become 
professional intermediaries. This person is assumed to be comfortable using 
computers, but not necessarily trained in their use. In addition, it is ^ 
assumejd T:hat* this user is a serious, well-intentioned searcher who is trying 
to u^e the system to solve a problem. The IIDA software and <|i a g nost i c 
procedures wer€~created *to help remove the barriers to access for these 
users. Minor modifications of the system, however > could make it available to 
a much wider audience. / v 

Given the nature of IIDA, it, is> not possible to completely separate 
the performance of IIDA from the performance of IIDA users. For summative * , 
evaluation purposes, the users of 'primary interest are those described in the 
third-stagjg v of IIDA evaluation studies. .Looking first at the baseline study^, 
the average^rat^d satisfaction of the IIDA ue^rs and the participants whp had 
intermediated searching done did not differ. In addition, the majority of the 
users in both' groups were pleased enough with their results and experience 
that they were willing to, recommend the system to their friends. 

In thfe judgment by the'end'user of the percentage of useful 
references retrieved in the searches, there was no significant difference 
fretween the results obtained by the professional searcher and those obtained 
by thq user performing his own search. It\was alsp found that there were no 
s^pificant differences in the number of IIDA detected errors or faults 
betyeen the tlDA trained and assisted searchers and the professional searchers 
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J. m Similarly, the two user groups which differed in the metho'd of 
training prior to doing IIDA assisted searching had no appreciable differences 
either on the . diagnostic measures or in the percentage of^useful references 
.retrieved. In addition, the members of both of these groups did not differ 
significantly from e^ctT other on several measures of the reactions to the?* 
search system, with, for example, most of the participants feeling they would 
recommend the system. to their friends. .Where the.two groups did ^differ, 
however, was in the evaluation of fhe system instructions and fcher ease of use 

of the system, m both cases the conventionally trained users indicated a 

■«% * 

higher degree 'of agreement than did the IIDA trained users. That is, they 
were more likely to feel that the instructions were e^sy to follow and more 
likely to indicate that the system was easier to use than they had expected. 

The finding of so few significant differences, 1 particularly on the 
diagnostic measures came as somewhat of a^urprise. 'In a case where no 
significant differences exist, the ,situati9n is always troublesome in that 
such a finding could occur because the two groups did v not "differ, or because * 
the'^measures chosen were irrelevant. This latter explanation, however , seems 
to be less reasonable than the idea that th^ system works as i£ was designed 
to work. One reason for arguing that the training and assistance routines 
worked as intended lies in the fact that in the baseline study the end user 
evaluation of the utility of the information retrieved in the searches did not 
differ significantly between the two groups. This means that a, group of 
individuals who had never before' done online searching were able, with IIDA 
training and^ assistance, to do searches which produced satisfactory results 
containing useful information, A second reason for not assuming that the' 
diagnostics are irrelevant as indie ies of searching behavior lies in the fact 
.that the diagnostics were all empirically developed. That is, they were all 
designed to .index and deal with problems encountered by searchers which have 
been either observed by the designers, reported in the literature by others,, 
or both. * * J 

'However, recognizing that the conclusions about the effectiveness of 
the diagnostics are not as clear and unambiguous as might be desirable, there 
are some additional directions for future research whicfc ought to be , 
fallowed. The first of these has to do with the evaluation of the. * 
.diagnostics. „The overall requirements for evaluation of the project' as a 
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^hole, as well as the available resources, dictated that some studies have a 
higher priority th^n others. This led to a concentration on the evaluation of 
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the IIDA system as a system rather than a more direct focus on the diagnostic 
component of the system.. future/evaluation devoted solely to the question 
of the ge!i|^o nuance of the diagnostic system, with and without various kinds of 
training in advance, would be. both useful and desirable. 1 

Another line of investigation dealing with the diagnostics which 
could be followed has to do with the possibility of - adaptation to user skill 

J level. The diagnostics were' designed for use in a limited context,* the 
training and assistance of the kind of .novice user described earlier and in 
the user ^studies reported in\he third-stage evaluation. There was some 1 
indication in the experiences of the professional searchers in the baseline 

0 study that wfren some of the diagnostics are employed with highly experienced 
se^r v chers, they may not b§ the most appropriate procedure!^ There may have 
J been times whfen .<H.dgnostics were triggered by relatively sophisticated 
searcher behavior which is appropriate in one^tfoiT^ext, but not in the 
situations with which IIDA was designed to cope. Thus, there is reason to 
believe that some of tjie diagnostics may not be equally useful or desirable 
for all user skill ^levels.. Consequently, the diagnostics should be tested out 
more thoroughly with persons of differing skill levels. In addition, it would 
also be desirable to, test different variations of,. the diagnostics. In 
particular it would be of interest to determine whether or not it is possible, 
by varying the thresholds in the existing set of diagnostics, as a function of 
the prior performance of the user, to get the diagnostics to perform 

adequately with individuals of different skill levels. That is, can 'they be 

n ... * a 

made adaptive? .One possibility is discussed briefly in section 9* 

» 

A third direction for future research has to do with the type of IIDA 
user. Although the original target user group for IIDA consisted of 
technically trained individuals interested in a particular class of search 
problems, there now seems to be no reason not to attempt tq extend IIDA. The 
technically or scientifically trained user may be only* one of several kinds of 
users who would find IIDA attractive and useful. In particular,, it sterns 
desirable to determine whether or not a system such as IIDA can be used to 

provide direct access to data bases for a wide variety of possible end users 

* * « 

interested in a wide variety of search problems. In fact, it may well be' that 
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IIDA, or an IIDA-like system, is a viable .method of introducing the computer 
and online searching to users with little or no background or experience with 
computers • « 

• - * 

%he fourth recommendation f or ^future research follows from one of the 
limitations of the studies described above* These studies, and many other 
studies^on searcher behavior, tend to be flawed by the fact th^t there is a. 
relatively short amount of time between when the participants in the testing 
are first exposed to the system .and when the evaluation of their performance, 
or of the system performance, takes place* In addition, there seems to have 
been little or no testing of information system users at several points over a 
long -period *of time in the investigation of development or change in behavior 
over time. In the studies described above, the volunteer subjects generally 
tended to want to do the whole set of exercises and then do their. searching in 
a relatively short time* It, seems particularly desirable that a long range 
study be done, over a period of a year or so, in which the focus, is upon how 
different people adapt to a new system,' how quickly they adapt, how their 
behavior changes over time, and how it changes as the result of multiple 
search experiences*. 

In conclusion, it seems that a new idea has been fairly tested in the 
very environment for which the concept was intended and has come off quite 
well. Indeed, one of the important characteristics of the £wo third-stajge 
evaluation studies is that there, appeared to be no differences among the 
results produced by the various user groups when there was every reason to 
expect in advance that there would be a ntimber of differences. While some 
aspefcts of these results are not entirely conclusive, they are supportive of 
the idea that the IIDA diagnostic procedures did indeed adequately measure 
important aspects of user performance. What is more certain, however, is that 
the IIDA system\represents a way of training and assisting novice users irrr* 
doing their own data base searching. Furthermore, IIDA clearly represents a 
viable alternative route to gaining direct data base access for those e^id 
users who cannot or wiLl not do more conventional forms of search training. 
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9. SEARCH CHARACTERISTICS 

;. - n 

Near the end of the project, a brief study was i^ndertaken to 
determine if there were useful patterns of searcher behavior in datd 
descriptive of frequency of use of various command types or of the probability 

v i 

of transition from one command type to another. Frequency of use ^ata appears 
to be descriptive of rather b f roadly based stylistic behavior, rather than. of 
specific errors or instances of poor usage. Hence x while we anticipate that 
* the overall frequency data may prove useful* in IIDA-like systems, we do not 
suggest that it is a replacement for the" types of diagnostic procedures 
employed in our m^in study, Rather, we see the investigation of frequencies^ 
pf use and transitional probabilities as a atep beyond the level of analysis 
incorporated in tfce diagnostics. The basic concept for this, kind of analysis 
in this context was set fortlT by P^nniman (19) and the results summarized here 
were reported by Chapman UO). While the result* presented here are far from* 
conclusive, they suggest new avenues of research which might lead to more 
sophisticated intermediary systems, better able to diagnose the nuances of 
user performance and hence batter able to respond to a particular levels of 
behavior. . 

9. 1 A Search as a Markov Process 

Penniman described* a search as a Markov chain of command states, 
where a state is the condition of having issued a command of a particular 
type. Markov chains can be analyzed at level* zero through any arbitrary 
number, jn* A level indicates the number of states preceding the one being 
considered; level zero gives the raw frequency of otcurrence of a state, 
level l'gives the frequencies of occurrence -of transitions from one state to 
another, with no intervening states; level 2 gives the frequencies of 
occurrences of transitions from state s^ to to* s^, etc*, Thus, level 
zero measures overall frequency of occurrence of each state, level one 
measures frequency of going from any state to any other state, level 2 the 
frequency of going from any state to any other via one intervening state, etc. 

Chapman applied Markovian analysis both to individual state 
occurrences §nd to strings, treating a sequence of commands of type N:. as a 



; single entity, a string of type- t. Data used in her analyses consisted of the 
search transcripts bf the IIDA evaluation studies involving (1) Drexel library 
science'graduate students, ' (2) Exxon engineers, and (3) Exxon scientists, m 



each case,- the subjects were divided ij^cJUrfr'oups and comparisons were made 
between them. In each case Significant differences were found between the two 
groups. ' « . 

•> 

The library science group were trained conventionally in the 
classroom. Then, some'of the participants performed searches "with, and some 
without, IIDA assistance. Those participants using IIDA assistance used 
' significantly ntore dictionary search ( expand ) and significantly fewer tvjje' * 
commands than the non-IIDA participants.* State transition data, at various' 
levels, were, reflective ot this basic fact. All significant differences in 
transition frequencies* involved 'those' two command types. String type 
transition analyses showed the same pattern as state transitions—significant 
differences in-- the use of expand and type commands. 

A second group consisted of engineers at Exxon Research and 
Engineering Company. Half of them were trained by IIDA and then performed a - 
search using IIDA assistance. Half took their search topics to professional 
searchers who performed the searches for them. Comparisons were made ^between 
the IIDA assisted users and the professionals, in this comparison there were 
significant differences in the zero order analysis for such co^oandsA limit 
and advanced usages of select (more by the professionals) and expand and tyjoe 
(more use by the IIDA users), it should be noted that an analysis of 
diagnostics between these two groups showed little difference, while the. 
command frequency analysis shows several differences. 

*** » 

The third group consisted of scientists at Exxon, half of whom were 
trained by IIDA and. half by conventional means. Then, both sets of 
participants searched using IIDA assistance. ^ These participants showed a 
significant difference in the use of advanced select commands, -with the IIDA 
trained searchers having the lower frequency. This difference is probably 
attributable to differing emphases in the training programs and suggests' that 
novice users are highly receptive to the searching style presented in the 
instructional material. * 

1 
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The diagnostics which were described earlier, and which were -the 
principal component of IIDA, detect quite specific errors or poor usage on the 
part of a searcher* The newer, Markovian measures are of general tendencies 
in the use of language* The two different measures appear to have different 
uses* Violation of an IIDA rule is a precise, correctable behavior* While we 
are not yet su**e how to interpret. deviation from the nofcm of a transition * 
frequency distribution, we believe that such a distribution may represent a 
measure of general skill^or sophistication in tfefe use of a query language* 
This is yet to be proven conclusively* 

One possible way of using the two forms of analysis together is that 
the Markovian analysis may be useful to provide a context which indicates to 
the system how the diagnostic information should be interpreted and acted 
upon* ' For example, «.if a future 'IIDA were to attempt to work, with a broad^ * 
range of users, certain diagnostic detectable behavior may be acceptable from 
a sophisticated U3er, but not from a novice* - Use of the Marker analysis, in 
conjunction with- the diagnostics, might provide a way of customizing, or 
fine-tuning, the system to the user* .Perhaps the best example of this is the 
IIDA imposed restriction on the amounj/^fi typing done by a user* If a novice, 
is searching, wd tried to limit the amount of typing because it had been our 
* experience that inexperienced searchers often type excessive numbers of 
records to avoid the intellectual work of using set definitions to reduce the h 
size of sets* If, however, an experienced searcher exceeds the IIDA imposed 
limit, other explanations for exceeding the limij: are both reasonable and 
probable* Thus, it would be nice to have the system sensitive enough to 
either relax the limit for more sophisticated searchers or not even send the 

searcher a message^at all* 

i • 

In the .comparisons of the IIDA searchers with, the Exxon 
professionals, the diagnostics showed no differences between the two groups* 
In addition, the end users 1 assessment of the value of the citations retrieved 
did not differ* However, the Markovian analysis did show a difference between 
the two groups* This suggests the possibility that the differences between 
the two groups were stylistic, reflecting the sophistication of the 
professional searchers;' If this is the case then the possibility exista that 
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in some cases events that were recorded by the diagnostics as having been 
"errors" might well have been "errors of experience." .These errors of 
experience being the kind of errors that result from knowing which principles 
or rules can be broken and when it is safe to do so. 



9.^ Other Characteristics of a Search 

♦Further analyses of state probability data are planned in an attempt 
to discover a basis for classifying entire searches into^raajor segments 
according to the pattern of usage of commands. The objective is to ehable an 
intelligent program to perceive what kind of user it is dealing with and to 
modify its own behdvioA accordingly. We do not expect that these new mea&irds 
will replace command diagnostics. These further analyses are not part -of the 
IIDA project, but we plan tc use our IIDA data to test some of them out. 

~ J. , 

One procedure ,is to organize command probability data by rank order, - 
following Zipf's work (21) with natural language. The command rank order 
distribution can be expected to follow approximately a negative exponential 
curve, as does natural language/ with allowances for the small size of the < 
samples (i.e, 10-20 commands, typically, compared with thousands of natural 
language words). On a semi-log scale this becomes a line whose slope and 
intercept then might -be used to characterize the distribution. It is hoped 
that such characterization of searches might provide for accurate 
characterization of users by type. * % 



Yet another measure to investigate, also suggested by Benniman, is 
Shannon's H (22) is applied to sta^e probabilities. These investigations will 
be pursued after the conclusion of the IIDA project. 
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10. ' THE FUTURE OF COMPTER ASSISTANTS 

T^ 1 



In "this brief section we look into the future ariS try to project the 
role for systems like IIDA. IIDA is not the only system of its generic type ' 
and our projection is based. on th'e future of the cfass, not the individual 
program* A commercial version of IIDA is being developed by the Franklin 
Institute Research 'Laboratories, .Inc. , of Philadelphia which organization 
participated in IIDA, project. The remainder of this section is devoted to 
^a candidly subjective appraisal of the future potential of these systems. 

We see IIDA as representative of a class of compute systems or 
system components whose purpose is to assist people in using other systems or 
components. Whether the assistant is ,a stand-alone program coupled with 
another stand-alone program (as* are IIDA and CONIT), or a* component designed 
to make the parent system easier to use, it is fast becoming recognized among 
designers of information systems that the computer, itself, can play a major 
irole in assisting people to u?e it to fullest advantage. / 

10.1, Matching Users and Programs 

Although designers of computer .software systems such as FORTRAN, 
PL/I, DIALOG, or ORBIT' do not customarily publish a statement defining the 
intended user group, experience with such systems leaves us with the 
impressions that: * 

1. Users, are expected either to have formal draining prior to us6 or 
to work through a professional intermediary f who is trained in 
system use and will perform the actual mechanical work for the 
end user. ~ ^ ■ «* 

i 

2; In the case of information retrieval systems, not a great range 
# of skill is expected. By comparison, general purpose programming 

langu^es have useful language subsets for -the novice* as well as 
special f eatures^that only the most skilled programmers use. 
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3, The proportion of the population who understand computers ' 
increases steadily and those who basically understand' computers 
and yho haVe mastered one language for dealing with them- may feel- 



impatient if they are required to go through a formal training 
course in order to program a simple. task" in a new language* 

10.2 The Role ,of Intermediary Programs • *» ^ 

We might, then divide the world of computer software, in terms of 
user interaction, into three parts: (1) the basic, work ^producing systems, 
such as applications programs, information retrieval programs or statistical 
analysis programs; t2) the language processors through which users deal with . 
the Type 1 programs^ and (3) "front end" programs designed to enable people to 
use a program of Type 1 or 2« 

Type 1 or 2 programs are typically "capital" — large, general 
purpose* If they were designed to be used with a Type 3 program-- as modern 
computers are designed to be used with an operating system, unlike their older 
predecessors — their designers could concentrate on what the system can do , not 
on how to, use it. Type -3 interface programs would take on the .burden of ^ 
enabling people to use the systems, and we can expect they will evolve with 
the capabilities- tot — -_- ' 
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li' Effectively customize the Type 1 or 2 programs to jsuit the needs 

of the organization or individual .using them* For example, if an 

organization's staff searches only one data base, an interface 
j 

assistant would employ a user language particularized to that 

\* 

data base* On' the other hand, CONIT represents an opposite 
.approach* An organization using it could elect to use a common 
language foi; a wide range of search services and the intermediary 
computer performs the translation 'into the particular language of 
each service. in a user-transparent manner* 6 

» • 

2* Enable people to make meaningful use of Type, 1 and 2 'systems with 

" - . . r 

only a minimum of prior training, and. to adapt continuously to 

increased levels of user* skills, thus enabling them to use as 

• "^mucK of the target system's capabilities as they can at their 

individual states of training* 



3.*' Serve as 'a focus for the local organizations training program in 
computer use and for assessment by the institution of how well ' ^ 
its information systems are being used, * 

10.3 Impact of Intermediary Computers * < 

As this concept, the customized intermediary assistant, Incomes . 
common, we can expect some changes in the way in which information systems ^are 
used locally and the way in which Type 1 and 2 programs are designed* and 
marketed; v \ 



1. The potential/user group, or market, for many kinds of programs - 
will increase, including those* now considered too technical or 
specialized for "general" users. For example, a system such as 
SPSS (Statistical Package *a#or the Social Sciences) cxfti^d be used 
by almost anyone who understands the statistics to perform « 
analyses, fhere would be no need to learn the specifics 

laying out records of input data, and the intermediary would help 

... * t * 

in determining which SPSS facility is best to emplpy for a given. 

problem." The user would concentrate on statistics , .not 

•programming. ^ 



2. Designers and vendors of these programs can concentrate on their 
roles as capital tool -producers and not on their role of 
fulfilling consumer needs. In many cases this can lead to more 

* , effective or efficient programs, as the large, centrally located 
programs are designed to 'process data efficiently and the slowex; 
user interface' operations are carried out in local mini- or 
microcomputers . 

3. More users will be at tf acted to use computers as the intimidating 
technical learning requirement lessens. We do not equate 
increased computer use necessarily with "good, 11 but we do feel 
that many 'information systems are 'underutilized. Indeed, IIDA 
was originally conceived iii response to a recognition that this 

. was true in the restricted area of use of scientific and 
technical bibliographic , information systems. * 

* . 65 
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4. These intermediary computers have the capability to serve as- 
multrplexoi^or concentrators (23), permitting several users to 
work simultaneously, sharing a single common line from the 
mini-computer po the target system. In the case of bibliographic 
searching, *we estimate that four or five relatively inexperienced 
searchers can share a single line to a search service, especially 
if they work at 300 baud terminal* while the intermediary-search 
service line ^operates at 1200 baud. While this would ^ppear to 
be a great savirig^for -users, the reality) will certainly be that 
search services will change their price schedules. Still, the 
net effect can be a more efficient operation for both user and 
seller, hence* less costly operation for both. One implication 
► may be that information retrieval program^ (^pe \) may have to 

* * be redesigned around rhe expectation of heavy shared use of 1 
c.ommunication lines. 8 



* In sypimary, we expect to see the customized intermediary computer 
become the common Way to? use large, complex, .capital type programs. Tfois will 
^result bott/in more us&rs of the latter systems and lower cost use. Both 
these effects—increasing breadth of use' and decreasing cost — are typical of, 
the ef fects ,of major changes in computer software over the years. 
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